Skip to content

feat: allow span filtering in service graph processor - #6453

Merged
javiermolinar merged 17 commits into
mainfrom
tempo-filter-service-graphs
Feb 17, 2026
Merged

feat: allow span filtering in service graph processor#6453
javiermolinar merged 17 commits into
mainfrom
tempo-filter-service-graphs

Conversation

@javiermolinar

@javiermolinar javiermolinar commented Feb 13, 2026

Copy link
Copy Markdown
Contributor

What this PR does:
This PR introduces span filtering for the Service Graph processor to prevent the creation of unwanted edges.
Ideally the filters should operate symetrically, discarding the clients and server spans. But since this is heavily depdendent on the intrumentation, it implements a best-effort cache to maintain graph integrity and prevent data skew.

Why:
Currently, the processor accepts every span, which can lead to cluttered or uncontrolled graphs. Some users generate service graphs from external sources (e.g., Grafana Beyla) and need the ability to disable graph generation for specific services within Tempo.

How:
The implementation reuses the spanfilter logic from the spanmetrics processor. To address the risk of creating "orphan edges" (where one side of a connection is filtered but the other isn't), a bounded cache tracks dropped spans:

  1. Every incoming span is evaluated against the filter.
  2. If a span is rejected, its key and side (client/server) is added to a "dropped spans" cache (a map with a TTL and a maximum element limit).
  3. Before creating a new edge, the processor checks this cache. If the peer span exists in the cache, the edge is discarded.
  4. TTL expiration and eviction are handled within the existing Expiration goroutine to minimize overhead.

Two new metrics are added:

metrics_generator_processor_service_graphs_dropped_edges_total: increments when an edge is dropped
metrics_generator_processor_service_graphs_dropped_span_side_cache_overflow_total: it increments every time it tries to add a new span to the cache when the max items is reached

Some caveats:

  • For simplicity, this cache reuses the existing edge store settings for TTL and max_items. We can monitor the new metrics to determine if independent configuration is required later.

  • If the cache is full, new "dropped" markers won't be added until existing items expire. This happens every two seconds. This is a deliberate trade-off to prioritize performance over perfect accuracy.

How it looks in action:

Before filtering
Screenshot 2026-02-17 at 12 48 57

After adding the following filter:

processor:
 service_graphs:
   filter_policies:
     - exclude:
       match_type: strict
       attributes:
         - key: resource.service.name
           value: shop-backend        
Screenshot 2026-02-17 at 12 39 54

The filter only removes the spans specifically tagged with that service name.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
@github-actions

github-actions Bot commented Feb 13, 2026

Copy link
Copy Markdown
Contributor

💻 Deploy preview deleted (feat: allow span filtering in service graph processor).

@javiermolinar

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e126cacee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modules/generator/processor/servicegraphs/servicegraphs.go
Comment thread docs/sources/tempo/configuration/manifest.md
Comment thread modules/generator/processor/servicegraphs/config.go
Comment thread modules/generator/processor/servicegraphs/servicegraphs.go
Comment thread modules/generator/processor/servicegraphs/store/store.go
Comment thread modules/overrides/userconfigurable/api/limits_test.go
Comment thread modules/overrides/userconfigurable/api/limits.go
Comment thread modules/generator/processor/servicegraphs/servicegraphs.go
Comment thread docs/sources/tempo/configuration/_index.md Outdated
Comment thread docs/sources/tempo/configuration/_index.md Outdated
javiermolinar and others added 3 commits February 17, 2026 13:04
…figurable-overrides.md

Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Comment thread modules/generator/processor/servicegraphs/store/store_test.go
Comment thread modules/generator/processor/servicegraphs/servicegraphs_test.go
Comment thread modules/generator/processor/servicegraphs/servicegraphs_test.go

@electron0zero electron0zero left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few nitpicks, otherwise lgtm.

…figurable-overrides.md

Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
@javiermolinar
javiermolinar merged commit b9f3aa3 into main Feb 17, 2026
25 checks passed
@javiermolinar
javiermolinar deleted the tempo-filter-service-graphs branch February 17, 2026 14:46
mattdurham pushed a commit to mattdurham/tempo that referenced this pull request Jun 18, 2026
* added spanfilter to filter out spans

* return the error if the filters are not correctly formatted

* add filters to the userconfigurable overrides

* add core logic for drop items cache

* wired up everything

* some tweaks

* refactor

* fmt and manifest

* tighten names

* rename metric

* drop already buffered counterparts when a span is filter out

* add a new test and documentation

* Update docs/sources/tempo/operations/manage-advanced-systems/user-configurable-overrides.md

Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>

* Update docs/sources/tempo/configuration/_index.md

Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>

* Update docs/sources/tempo/configuration/_index.md

Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>

* bail early when the span kind is not client or server

* Update docs/sources/tempo/operations/manage-advanced-systems/user-configurable-overrides.md

Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>

---------

Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants