feat(tools): Querytee Goldfish #17959
Open
+9,728
−23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements a new query sampling and analysis system called Goldfish that integrates with the existing QueryTee proxy to capture performance metrics and compare responses between backend cells. Key features: - Configurable tenant-based query sampling with random rate limiting - Privacy-compliant performance statistics extraction (no raw log content) - Response integrity verification using FNV32 content hashing - Asynchronous processing to avoid blocking client responses - CloudSQL storage backend for captured metrics and comparisons - Comprehensive Prometheus metrics for monitoring sampling decisions - Backward compatible - existing QueryTee functionality unchanged The implementation extracts query performance statistics including execution time, queue time, bytes/lines processed, and processing rates from Loki responses. All sensitive data is excluded from storage, with only performance metrics and content hashes retained for analysis. Goldfish operates as an optional feature behind configuration flags and maintains complete isolation from existing QueryTee comparison logic.
Add experimental warning to make it clear that the API and configuration may change in future releases.
- Replace strings.SplitSeq with standard strings.Split for better readability - Fix response body restoration to properly recreate readable body with bytes.NewReader - Add missing bytes import to manager.go
JordanRushing
force-pushed
the
querytee-goldfish
branch
from
b77baca to
725ee66
Compare
- Update CloudSQL configuration to work with CloudSQL proxy sidecar pattern - Add host, port, user, and password fields for proxy connection - Replace pgx driver with lib/pq for PostgreSQL connectivity - Default CloudSQL proxy host to 'cloudsql-proxy' on port 5432 - Remove CloudSQL connection name in favor of standard database connection parameters This implementation supports the typical CloudSQL deployment pattern where a proxy sidecar handles authentication and SSL/TLS connections.
Signed-off-by: Jordan Rushing <rushing.jordan@gmail.com>
- Add POST method support for query endpoints alongside GET - Fix critical goroutine variable capture bug causing race conditions - Fix race condition in sampler by protecting config access with mutex - Add context timeout for async Goldfish processing to prevent hangs - Enhance logging to show query comparison stats without sensitive data - Log execution times, bytes processed, and entries returned for both cells - Add performance ratio logging and significant difference detection - Improve backend selection logic to properly identify Cell A/B - Add debug logging for sampling decisions and Goldfish attachment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR introduces Goldfish, an experimental feature for QueryTee that enables
privacy-compliant query sampling and comparison between multiple Loki cells.
Motivation
When running multiple Loki cells (e.g., during migrations or A/B testing), operators
need visibility into response differences and performance variations between cells.
Goldfish addresses this by sampling queries and comparing their responses without
storing sensitive log content.
Key Features
tenants)
processed
Implementation
Goldfish integrates seamlessly with QueryTee's existing proxy flow:
Configuration Example
Testing
This is marked as an experimental feature with the understanding that APIs may evolve
based on operational feedback.
go mod tidy && go mod vendorwere run for DB dependencies.Which issue(s) this PR fixes:
N/A
Special notes for your reviewer:
N/A
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mddeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR