Skip to content

fix(querytee): Fix TestProxyEndpoint_QuerySplitting clock-drift bug - #21779

Merged
jnewbigin merged 1 commit into
mainfrom
jnewbigin/flakey-test
May 8, 2026
Merged

fix(querytee): Fix TestProxyEndpoint_QuerySplitting clock-drift bug#21779
jnewbigin merged 1 commit into
mainfrom
jnewbigin/flakey-test

Conversation

@jnewbigin

@jnewbigin jnewbigin commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • TestProxyEndpoint_QuerySplitting was flaking in CI with 0/3 instead of 2/1 for the old/recent query split classification.
  • Root cause: the test captures threshold = time.Now().Truncate(time.Minute) - 3h once at setup, but the engine router computes v2End = time.Now() - StorageLag per request. For logs queries, alignV2Range is a no-op (step=0), so v2End drifts second-by-second with wall-clock. When the truncation landed late in a minute and the prior 2-second sleep pushed total drift past 60s, v2End > threshold + 60s and all three queries got misclassified as "recent".
  • Fix is test-only: classify queries by whether their end matches the request's end (the post-v2 split runs to request end; the v2 split ends earlier) instead of an absolute boundary, and widen the mock server's split boundary from threshold + 60s to threshold + 30m so it stays stable under any plausible drift.

Test plan

  • go test -tags=assert -count=10 -run TestProxyEndpoint_QuerySplitting ./pkg/querytee/... passes (10/10)
  • go test -tags=assert ./pkg/querytee/... passes

Note

Low Risk
Low risk because changes are confined to test logic and only adjust how the mock split boundary and assertions are computed to avoid time.Now() drift flakes.

Overview
Fixes CI flakiness in TestProxyEndpoint_QuerySplitting by making the mock backend’s split boundary drift-tolerant (using threshold + 30m and treating equality as recent).

Updates assertions to classify old vs recent sub-queries based on whether the query end matches the original request end (instead of comparing against a step-aligned absolute boundary), and removes the now-unused stepAlignUp helper.

Reviewed by Cursor Bugbot for commit b0ddf34. Bugbot is set up for automated code reviews on this repo. Configure here.

The test captured `threshold = time.Now().Truncate(time.Minute) - 3h` once
at setup, then asserted v2-split end times fell within `threshold + 60s`.
But the engine router computes `v2End = time.Now() - StorageLag` per
request, and for logs queries `alignV2Range` is a no-op (step=0), so v2End
drifts second-by-second with wall-clock. When the truncation landed late
in a minute and the prior 2-second sleep pushed total drift past 60s, all
three queries were misclassified as "recent" — the 0/3 failure seen in CI.

Classify queries by whether their end matches the request's end (post-v2
split runs to request end; v2 split ends earlier) instead of an absolute
boundary, and widen the mock server's split boundary to threshold + 30m
so it stays stable under any plausible drift.
@jnewbigin
jnewbigin requested a review from a team as a code owner May 7, 2026 00:56
@jnewbigin jnewbigin changed the title fix(querytee): de-flake TestProxyEndpoint_QuerySplitting clock-drift bug May 7, 2026
@jnewbigin
jnewbigin merged commit da5a172 into main May 8, 2026
86 of 87 checks passed
@jnewbigin
jnewbigin deleted the jnewbigin/flakey-test branch May 8, 2026 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 participants