[Infra] Post-enrichment filtering for Hosts exclusion filters by rmyz · Pull Request #260426 · elastic/kibana

rmyz · 2026-03-31T10:05:38Z

Summary

Hosts that should be excluded by negative filters (e.g., NOT cloud.provider: gcp) were still appearing in the Hosts UI. This happened because APM documents for some hosts lack the filtered field entirely, which causes Elasticsearch must_not clauses to pass those documents through. After enrichment (getAllHosts), these hosts would show the excluded metadata value (e.g., cloud.provider: gcp), contradicting the user's filter.

Changes

Post-enrichment filter in getHosts: After getAllHosts enriches host data with metadata (cloud.provider, host.os.name, host.ip), a new filter checks each host's metadata against must_not exclusion values from the user's query. Hosts whose enriched metadata matches an excluded value are removed from the final result.
New extractExcludedMetadataValues utility: Parses must_not clauses from the query and extracts excluded field/value pairs for metadata fields. Supports match_phrase, match, term, terms, and nested bool.should inside must_not.
Aligned getHostsCount: The count endpoint now follows the same pattern — gets the host name union, and when there are excluded metadata values, runs a lightweight metadata-only query to post-filter before counting.
Simplified getFilteredHostNames: Returns string[] instead of { allHosts, filteredHosts }. The allHosts sub-aggregation was removed since it's no longer needed.
Simplified getHostNames: Reverted to a plain union of infra + APM hosts. The previous infra-based exclusion logic was insufficient for APM-only hosts not recognized by infra.
Removed getHasDataFromSystemIntegration call: The pre-check was unnecessary since getFilteredHostNames handles empty results gracefully.
Extracted MAX_HOST_COUNT_LIMIT to common/constants.ts instead of hardcoding.
Removed queryHasNegation utility (no longer needed).

Demo

Before

Kapture.2026-03-31.at.15.37.33.mp4

After

Kapture.2026-03-31.at.12.43.39.mp4

How it works

User filter: NOT cloud.provider: gcp

1. getHostNames: union of filteredHosts (infra) + apmHosts → candidate list
2. getAllHosts: enriches candidates with metadata (cloud.provider, host.os.name, host.ip)
3. extractExcludedMetadataValues: parses must_not → { 'cloud.provider': Set(['gcp']) }
4. Post-filter: removes any host where metadata matches excluded values
   e.g., host with enriched cloud.provider: gcp → removed

Testing

8 unit tests for getHosts covering union behavior and post-enrichment filtering (including the exact reproduction scenario)
9 unit tests for extractExcludedMetadataValues covering all query clause types
All existing tests updated and passing

Hosts that should be excluded by negative filters (e.g., NOT cloud.provider: gcp) were still appearing because APM documents often lack the filtered field, allowing them to bypass Elasticsearch must_not clauses. This adds a post-enrichment filter that checks enriched metadata values against the exclusion criteria after data retrieval, catching hosts that slip through earlier query-level filters. Closes elastic#256157 Made-with: Cursor

rmyz · 2026-03-31T10:06:29Z

/ci

elasticmachine · 2026-03-31T10:51:08Z

💔 Build Failed

Buildkite Build
Commit: 51e2e41

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #128 / X-Pack Accessibility Tests - Group 1 Management index management indices indices with data index details index details - overview
[job] [logs] FTR Configs #128 / X-Pack Accessibility Tests - Group 1 Management index management indices indices with data index details index details - overview

Metrics [docs]

✅ unchanged

History

cc @rmyz

rmyz · 2026-03-31T14:11:04Z

/ci

Copilot

Pull request overview

Fixes Hosts UI negative-filter exclusions by post-filtering enriched host metadata so hosts whose APM docs lack the filtered field don’t “slip through” must_not and reappear after enrichment.

Changes:

Adds extractExcludedMetadataValues to parse must_not exclusions for enriched host metadata fields.
Applies post-enrichment filtering in getHosts to remove hosts whose enriched metadata matches excluded values.
Updates getHostsCount to align with the new behavior by post-filtering via a lightweight metadata aggregation.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/utils.ts	Adds utility to extract excluded metadata values from `must_not` query clauses.
x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/utils.test.ts	Adds unit tests for excluded-metadata extraction.
x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/host/get_hosts_count.ts	Aligns count logic with post-enrichment exclusion filtering using metadata aggregations.
x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/host/get_hosts.ts	Applies post-enrichment filtering to host results and simplifies host-name collection.
x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/host/get_hosts.test.ts	Adds unit tests covering union behavior and post-enrichment filtering scenarios.
x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/host/get_filtered_hosts.ts	Simplifies API to return `string[]` and adjusts query filter composition/agg naming.
x-pack/solutions/observability/plugins/infra/common/constants.ts	Extracts `MAX_HOST_COUNT_LIMIT` constant.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Wrap documentsFilter in nested bool inside filter to enforce minimum_should_match - Handle multi-valued metadata (host.ip) in post-enrichment filtering - Guard bool.should extraction with minimum_should_match check Made-with: Cursor

rmyz · 2026-03-31T14:36:32Z

/ci

…ction When must/filter clauses are present in a bool query, ES defaults minimum_should_match to 0, making should clauses optional. Only extract excluded values from bool.should when it is actually required. Made-with: Cursor

rmyz · 2026-04-01T06:58:10Z

/ci

elasticmachine · 2026-04-01T07:17:46Z

Pinging @elastic/obs-presentation-team (Team:obs-presentation)

macroscopeapp · 2026-04-01T07:18:29Z

Approvability

Verdict: Needs human review

This PR introduces post-enrichment filtering that changes which hosts are returned and counted based on metadata exclusion filters - a significant runtime behavior change affecting data processing pipelines. Additionally, the author does not own any of the modified files, which are all owned by @elastic/obs-presentation-team.

^{You can customize Macroscope's approvability policy. Learn more.}

sbelastic

code LGTM, also tested locally and everything worked :)

kibanamachine · 2026-04-01T09:11:45Z

Starting backport for target branches: 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/23841119069

…c#260426) (cherry picked from commit 705ddb9)

kibanamachine · 2026-04-01T09:20:24Z

💔 Some backports could not be created

Status	Branch	Result
❌	9.2	Backport failed because of merge conflicts
✅	9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 260426

Questions ?

Please refer to the Backport tool documentation

…c#260426) (cherry picked from commit 705ddb9) # Conflicts: # x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/host/get_filtered_hosts.ts # x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/host/get_hosts_count.ts

rmyz · 2026-04-01T09:51:34Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…260426) (#260649) # Backport This will backport the following commits from `main` to `9.3`: - [[Infra] Post-enrichment filtering for Hosts exclusion filters (#260426)](#260426)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Sergi Romeu <sergi.romeu@elastic.co>

…260426) (#260658) # Backport This will backport the following commits from `main` to `9.2`: - [[Infra] Post-enrichment filtering for Hosts exclusion filters (#260426)](#260426)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…c#260426)

rmyz added release_note:fix backport:skip This PR does not require backporting Team:obs-presentation Focus: APM UI, Infra UI, Hosts UI, Universal Profiling, Obs Overview and left Navigation labels Mar 31, 2026

rmyz self-assigned this Mar 31, 2026

Merge branch 'main' into fix/hosts-exclusion-filter-post-enrichment

51e2e41

rmyz changed the title ~~fix: [Infra] Post-enrichment filtering for Hosts exclusion filters~~ Mar 31, 2026

Merge branch 'main' into fix/hosts-exclusion-filter-post-enrichment

21a47de

rmyz requested a review from Copilot March 31, 2026 14:14

Copilot AI reviewed Mar 31, 2026

View reviewed changes

fix: address Copilot review comments

f1e3c7b

- Wrap documentsFilter in nested bool inside filter to enforce minimum_should_match - Handle multi-valued metadata (host.ip) in post-enrichment filtering - Guard bool.should extraction with minimum_should_match check Made-with: Cursor

macroscopeapp Bot reviewed Mar 31, 2026

View reviewed changes

Comment thread x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/utils.ts

rmyz added 2 commits April 1, 2026 08:56

Merge branch 'main' into fix/hosts-exclusion-filter-post-enrichment

3a9de4b

macroscopeapp Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread x-pack/solutions/observability/plugins/infra/server/routes/infra/lib/utils.ts

rmyz marked this pull request as ready for review April 1, 2026 07:17

rmyz requested a review from a team as a code owner April 1, 2026 07:17

rmyz added backport:version Backport to applied version labels v9.3.3 v9.2.8 and removed backport:skip This PR does not require backporting labels Apr 1, 2026

sbelastic approved these changes Apr 1, 2026

View reviewed changes

rmyz merged commit 705ddb9 into elastic:main Apr 1, 2026
30 checks passed

kibanamachine added the v9.4.0 label Apr 1, 2026

rmyz deleted the fix/hosts-exclusion-filter-post-enrichment branch April 1, 2026 09:16

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 1, 2026

[Infra] Post-enrichment filtering for Hosts exclusion filters (elasti…

bcfdc3d

…c#260426) (cherry picked from commit 705ddb9)

kibanamachine mentioned this pull request Apr 1, 2026

[9.3] [Infra] Post-enrichment filtering for Hosts exclusion filters (#260426) #260649

Merged

rmyz mentioned this pull request Apr 1, 2026

[9.2] [Infra] Post-enrichment filtering for Hosts exclusion filters (#260426) #260658

Merged

jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Apr 1, 2026

[Infra] Post-enrichment filtering for Hosts exclusion filters (elasti…

ddacc5b

…c#260426)

eokoneyo pushed a commit to davismcphee/kibana that referenced this pull request Apr 2, 2026

[Infra] Post-enrichment filtering for Hosts exclusion filters (elasti…

45b29b2

…c#260426)

paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Apr 2, 2026

[Infra] Post-enrichment filtering for Hosts exclusion filters (elasti…

9038272

…c#260426)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Infra] Post-enrichment filtering for Hosts exclusion filters#260426

[Infra] Post-enrichment filtering for Hosts exclusion filters#260426
rmyz merged 6 commits intoelastic:mainfrom
rmyz:fix/hosts-exclusion-filter-post-enrichment

rmyz commented Mar 31, 2026 •

edited by kibanamachine

Loading

rmyz commented Mar 31, 2026

elasticmachine commented Mar 31, 2026 •

edited

Loading

rmyz commented Mar 31, 2026

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rmyz commented Mar 31, 2026

Uh oh!

rmyz commented Apr 1, 2026

Uh oh!

elasticmachine commented Apr 1, 2026

macroscopeapp Bot commented Apr 1, 2026

sbelastic left a comment

Uh oh!

kibanamachine commented Apr 1, 2026

kibanamachine commented Apr 1, 2026

rmyz commented Apr 1, 2026

Labels

5 participants

Conversation

rmyz commented Mar 31, 2026 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Demo

Before

After

How it works

Testing

rmyz commented Mar 31, 2026

elasticmachine commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

History

rmyz commented Mar 31, 2026

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rmyz commented Mar 31, 2026

Uh oh!

rmyz commented Apr 1, 2026

Uh oh!

elasticmachine commented Apr 1, 2026

macroscopeapp Bot commented Apr 1, 2026

Approvability

sbelastic left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Apr 1, 2026

kibanamachine commented Apr 1, 2026

💔 Some backports could not be created

Manual backport

Questions ?

rmyz commented Apr 1, 2026

💚 All backports created successfully

Questions ?

Labels

5 participants

rmyz commented Mar 31, 2026 •

edited by kibanamachine

Loading

elasticmachine commented Mar 31, 2026 •

edited

Loading