Tempo query improve search performance

[pavolloffay]

Depends on grafana/tempo#4159

kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  timeout: 90m
  extraConfig:
    tempo:
      ingester:
        max_block_duration: 2m
  storage:
    secret:
      name: s3-secret
      type: s3
  storageSize: 5Gi
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
        findTracesConcurrentRequests: 30
        ingress:
          route:
            termination: reencrypt
          type: route
EOF

Generate data

docker run --rm -it --net=host ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest traces --otlp-http=true --otlp-insecure --otlp-endpoint localhost:4318 --child-spans=2 --workers=3  --traces=1000

[pavolloffay]

Defaulting to 1 might not be ideal. Perhaps we can default to numberofqueriers * 2

[pavolloffay]

I will keep the default to 1, here is some rationale:

Customers complain when querying large number of traces e.g. 500-1500. A single trace get from S3 can take from 2s to 8s. for 500 traces it is 500*4=2000s/60=33mins. Cutting the time in half with 2 concurrent requests does not bring any value, however 30 concurrent requests significantly improve the situation but for that customers need to scale up queriers.

[andreasgerstmayr]

I think we should use a default value which improves the current situation instead of leaving it as-is.

I like your suggestion of making the default value based on the number of queriers, maybe we can leave it 0 in the webhook and compute the default (2 * .spec.template.querier.replicas if set to 0) in the manifest generation?

[andreasgerstmayr]

My above comment is for TempoStack, for TempoMonolithic we can't scale up queriers, so maybe let's go with 2 if the default max concurent requests per querier is 20?

[pavolloffay]

I have changed the default for monolithic to 2 and tempostack to queriers*2.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 23.07692% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.14%. Comparing base (88d46a5) to head (88c172f).
⚠️ Report is 221 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/manifests/config/build.go	0.00%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1048      +/-   ##
==========================================
- Coverage   69.20%   69.14%   -0.06%     
==========================================
  Files         110      110              
  Lines        7049     7059      +10     
==========================================
+ Hits         4878     4881       +3     
- Misses       1881     1888       +7     
  Partials      290      290              

Flag	Coverage Δ
unittests	69.14% <23.07%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.