feat: Add option to run tempo-vulture in validation mode - #5605
Merged
Conversation
|
|
davidham
force-pushed
the
feature/add-option-to-run-once-and-exit
branch
from
September 4, 2025 19:04
93a50aa to
4f0331a
Compare
davidham
marked this pull request as ready for review
September 4, 2025 21:59
davidham
requested review from
carles-grafana,
electron0zero,
ie-pham,
javiermolinar,
joe-elliott,
mapno,
mdisibio,
ruslan-mikhailov,
stoewer,
yvrhdn and
zalegrala
as code owners
September 4, 2025 21:59
davidham
marked this pull request as draft
September 5, 2025 21:13
Contributor
Author
|
Changing back to draft so I can add basic auth. |
davidham
force-pushed
the
feature/add-option-to-run-once-and-exit
branch
from
September 9, 2025 15:10
5074b3d to
021cc90
Compare
davidham
marked this pull request as ready for review
September 9, 2025 15:33
zalegrala
reviewed
Sep 10, 2025
zalegrala
left a comment
zalegrala
left a comment
Contributor
There was a problem hiding this comment.
Overall this is looking reasonable. I left a few thoughts.
davidham
force-pushed
the
feature/add-option-to-run-once-and-exit
branch
from
September 10, 2025 22:23
021cc90 to
efa491b
Compare
zalegrala
reviewed
Sep 18, 2025
zalegrala
left a comment
zalegrala
left a comment
Contributor
There was a problem hiding this comment.
This is looking reasonable. I left a few more thoughts.
| ) int { | ||
| accessPolicyToken := os.Getenv("TEMPO_ACCESS_POLICY_TOKEN") | ||
| if accessPolicyToken == "" { | ||
| logger.Error("TEMPO_ACCESS_POLICY_TOKEN environment variable is required in validation mode") |
Contributor
There was a problem hiding this comment.
Consider making this optional for use and testing without auth.
- Use debug level in that debug function - return err instead of ctx.Err() - Update vs.sleepWithContext to use the vs.clock so it's faster in tests
Merged all of my new 'withContext' metjods with their originals, and updated all the callers to pass in context.Background() so it would compile
davidham
force-pushed
the
feature/add-option-to-run-once-and-exit
branch
from
September 29, 2025 21:04
a76462b to
79d00a3
Compare
zalegrala
approved these changes
Oct 8, 2025
zalegrala
left a comment
zalegrala
left a comment
Contributor
There was a problem hiding this comment.
This looks reasonable to me. Thanks for addressing some of the comments.
Merged
This was referenced Jan 21, 2026
Merged
mattdurham
pushed a commit
to mattdurham/tempo
that referenced
this pull request
Jun 18, 2026
* feat: Read/write validation working, simple search * feat: WIP on adding search for each trace * feat: exposed port 4317 * feat: Extracted write, read, search into separate methods * feat: Updated to use new service code * feat: Added MockClock; fixed an error in a prev mock * feat: Added tests for the three parts of the validation * feat: Added runValidation test * docs: Updated changelog * docs: CHANGELOG.md without Prettier LOL * fix: Updates for linter * fix: more linter changes * feat: Added new methods to create OTLP exporters with auth enabled Also updated old functions to call the new, for backward compatibility * feat: Added TempoBasicAuthToken to ValidationConfig * feat: Updated to use new auth-enabled versions * feat: Added EmitBatchesWithContext * feat: Moved validationConfig setup to its own function * chore: lowercase errors * feat: Added context-aware methods * feat: Added methods to mocks * fix: return 1 rather than exit * renamed sleep method * fix: linter issue * WIP on validateTraceSearch * feat: Set ingester rf to 3; formatting * feat: Test updates * fix: linter issues * fix: moar linter issues * feat: Made TEMPO_ACCESS_POLICY_TOKEN a const * feat: Also moved defer() statement * feat: PR comment fixes - Use debug level in that debug function - return err instead of ctx.Err() - Update vs.sleepWithContext to use the vs.clock so it's faster in tests * feat: Add constants for headers * feat: Refactor for withContext Merged all of my new 'withContext' metjods with their originals, and updated all the callers to pass in context.Background() so it would compile * feat: Update to use the new const * docs: Changelog formatting * feat: rename header
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
This PR adds a
validationmode to tempo-vulture that allows it to be used as a post-deployment validation test. It does similar work to tempo-vulture, but instead of running continuously and reporting its results as metrics, it will createntraces (determined by the--validation-cyclesflag) and attempt to read them back by ID, and query for them, and then exit. It will exit0if all operations succeed and exit1if any fail.I tested this against the
example/docker-compose/distributedDocker Compose file. The search validations did not pass but I intend to try them against our federal dev environment and see if they work there.Which issue(s) this PR fixes:
Relates to grafana/deployment_tools#294598.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]