feat: limit recent queries

[javiermolinar]

What this PR does:
This PR addresses an issue where queries using recent timestamps (especially "now") can return inaccurate or inconsistent results due to laggy replicas and incomplete data ingestion.

A new middleware operates transparently between the API layer and the queriers, ensuring that external interfaces remain unchanged while internal request processing becomes more robust and consistent.

This change ensures that Tempo queries return consistent, accurate results by avoiding the "dangerous zone" of very recent data that may be incomplete across replicas, while standardizing how time ranges are handled throughout the query pipeline.

Problem

When search and metrics queries use very recent end times (like time.Now()), they can hit incomplete data because:

• Laggy replicas: Some replicas may not have received the most recent data yet
• Ingestion delays: Data is still being processed and may not be immediately available
• Inconsistent results: Different replicas return different results for the same query

This leads to unreliable query results.

Solution

Introduces configurable time for clamping the end range:

• query_end_cutoff,: Prevents queries from using end times too close to "now" (defaults to 30s)

Key Behavioral Changes

1. All requests now include explicit start/end parameters: Previously, requests without time ranges would be forwarded to queriers as-is. Now, the frontend middleware ensures every request to queriers includes clamped start and end parameters.

2. "since" parameter handling: The search API's since parameter (e.g., since=1h) is now processed by the frontend and converted to explicit start and end parameters before forwarding to queriers

3. GRPC requests with empty start/end: GRPC requests that previously had default/empty start and end times are now resolved to explicit timestamp values during the clamping process, ensuring consistent behavior across all query types.

Implementation Details

• Pipeline Middleware: NewAdjustStartEndWare applies clamping to all incoming requests before they reach queriers
• Parameter Normalization: All relative time parameters (since) are converted to absolute timestamps

Test Updates

Updated integration tests to account for the new clamping behavior:

• Explicit Time Ranges: Tests now specify explicit start/end times instead of relying on defaults
• Proper Buffers: Added time buffers in tests to avoid hitting the clamped regions
• Timing Fixes: Resolved flaky tests caused by queries hitting incomplete recent data

Breaking Changes

None. This change is fully backward compatible:

• Client APIs: Still accept the same parameters (including since) - the processing is internal
• Querier Interfaces: Still receive the same parameter types, just with guaranteed populated values instead of potentially empty ones
• GRPC/HTTP APIs: All existing endpoints work exactly the same from a client perspective

Not a single line of business logic has changed

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[javiermolinar]

@joe-elliott I was able to make it work without the extra config DefaultQueryStart in fact I've simplified most of the tests configs. Could you take it another look?

[joe-elliott]

PR looking good close to merging. Can we go ahead and make the config change now? Also needs a changelog.

[joe-elliott]

same check 2x? remove the second one?

[javiermolinar]

It´s not really duplicated since the QueryBackendAfter can be different for search and metrics. Nevertheless we already discussed moving this property up. I will remove the second one

[joe-elliott]

oh, good call. i didn't notice. i'm good with the double check

[joe-elliott]

one cleanup request, but looks good otherwise