feat: allow span filtering in service graph processor - #6453
Merged
Conversation
Contributor
|
💻 Deploy preview deleted (feat: allow span filtering in service graph processor). |
javiermolinar
marked this pull request as ready for review
February 13, 2026 14:51
javiermolinar
requested review from
carles-grafana,
electron0zero,
ie-pham,
knylander-grafana,
mapno,
mattdurham,
mdisibio,
oleg-kozlyuk-grafana,
ruslan-mikhailov,
stoewer,
yvrhdn and
zalegrala
as code owners
February 13, 2026 14:51
Contributor
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e126cacee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…figurable-overrides.md Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
carles-grafana
approved these changes
Feb 17, 2026
electron0zero
approved these changes
Feb 17, 2026
electron0zero
left a comment
Member
There was a problem hiding this comment.
few nitpicks, otherwise lgtm.
…figurable-overrides.md Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
mattdurham
pushed a commit
to mattdurham/tempo
that referenced
this pull request
Jun 18, 2026
* added spanfilter to filter out spans * return the error if the filters are not correctly formatted * add filters to the userconfigurable overrides * add core logic for drop items cache * wired up everything * some tweaks * refactor * fmt and manifest * tighten names * rename metric * drop already buffered counterparts when a span is filter out * add a new test and documentation * Update docs/sources/tempo/operations/manage-advanced-systems/user-configurable-overrides.md Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com> * Update docs/sources/tempo/configuration/_index.md Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com> * Update docs/sources/tempo/configuration/_index.md Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com> * bail early when the span kind is not client or server * Update docs/sources/tempo/operations/manage-advanced-systems/user-configurable-overrides.md Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com> --------- Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
This PR introduces span filtering for the Service Graph processor to prevent the creation of unwanted edges.
Ideally the filters should operate symetrically, discarding the clients and server spans. But since this is heavily depdendent on the intrumentation, it implements a best-effort cache to maintain graph integrity and prevent data skew.
Why:
Currently, the processor accepts every span, which can lead to cluttered or uncontrolled graphs. Some users generate service graphs from external sources (e.g., Grafana Beyla) and need the ability to disable graph generation for specific services within Tempo.
How:
The implementation reuses the spanfilter logic from the spanmetrics processor. To address the risk of creating "orphan edges" (where one side of a connection is filtered but the other isn't), a bounded cache tracks dropped spans:
Two new metrics are added:
metrics_generator_processor_service_graphs_dropped_edges_total: increments when an edge is dropped
metrics_generator_processor_service_graphs_dropped_span_side_cache_overflow_total: it increments every time it tries to add a new span to the cache when the max items is reached
Some caveats:
For simplicity, this cache reuses the existing edge store settings for TTL and
max_items. We can monitor the new metrics to determine if independent configuration is required later.If the cache is full, new "dropped" markers won't be added until existing items expire. This happens every two seconds. This is a deliberate trade-off to prioritize performance over perfect accuracy.
How it looks in action:
Before filtering

After adding the following filter:
The filter only removes the spans specifically tagged with that service name.
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]