[ML] Enable alerts filtering with KQL bar by rbrtj · Pull Request #240100 · elastic/kibana

rbrtj · 2025-10-22T13:44:43Z

Resolves #136392
This PR adds alert filtering capability for the Anomaly Detection Alerting Rule using the KQL bar.
It provides proper auto-suggestions for the selected result and job type.

How to test:

1. Create a pre-configured Kibana sample e-commerce job without starting the datafeed.
2. Create an Anomaly Detection alerting rule based on the created job, e.g., with a 168h lookback interval and 168 latest buckets, select either the `record` or `influencer` result type.
3. Go to the AD jobs page and start the datafeed with `beginning of data` and `no end time`
4. Navigate back to the alerting rule, run the rule, and verify that it triggers an alert.
5. Create a new alerting rule with the desired filter. For example, if you alert was triggered for `customer_full_name.keyword: Eddie Lambert`, try filtering it out with: `NOT customer_full_name.keyword: "Eddie Lambert"`
6. Verify that the new rule does not trigger an alert for the specified `customer_full_name`
7. It's easier to use an email or slack connector to inspect the alerts.

elasticmachine · 2025-10-29T10:25:19Z

Pinging @elastic/ml-ui (:ml)

Copilot

Pull Request Overview

This PR adds KQL-based filtering capability to ML Anomaly Detection alerting rules, allowing users to filter which anomalies trigger alerts based on field values. The implementation provides autocomplete suggestions tailored to the selected job and result type, and automatically disables conflicting UI controls when users specify score or interim filters in the KQL expression.

Key Changes:

Added KQL filter input component with autocomplete for anomaly result fields
Implemented logic to detect when KQL filters conflict with UI controls (severity, interim results) and disable those controls accordingly
Extended server-side query logic to apply custom KQL filters when fetching anomaly results

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`alerting_service.ts`	Added KQL filter parsing and integration into Elasticsearch queries for anomaly detection alerts
`severity_control.tsx`	Added disabled state support with help text when KQL contains score filters
`register_anomaly_detection_rule.tsx`	Added KQL filter validation in rule registration
`ml_anomaly_alert_trigger.tsx`	Integrated KQL filter component and state management for field usage detection
`interim_results_control.tsx`	Added disabled state with tooltip when KQL contains is_interim filter
`get_relevant_anomaly_fields.ts`	Helper function to determine which anomaly fields are relevant for autocomplete based on job configuration and result type
`anomaly_kql_filter.tsx`	New component implementing KQL search bar with ML anomalies data view and filtered suggestions
`detect_anomaly_alert_field_usage.ts`	Utility to parse KQL and detect usage of score and interim fields
`alerts.ts`	Added customFilter parameter to alert params type definition
`alerts.ts` (constants)	Moved score field mapping to shared constant
`v1.ts`	Added customFilter to schema validation

x-pack/platform/plugins/shared/ml/server/lib/alerts/alerting_service.ts

...atform/plugins/shared/ml/public/alerting/anomaly_detection_rule/ml_anomaly_alert_trigger.tsx

github-actions · 2025-10-29T12:34:04Z

@rbrtj, it looks like you're updating the parameters for a rule type!

Please review the guidelines for making additive changes to rule type parameters and determine if your changes require an intermediate release.

peteharverson · 2025-10-31T12:15:56Z

...atform/plugins/shared/ml/public/alerting/anomaly_detection_rule/ml_anomaly_alert_trigger.tsx

+        }
+      } else {
+        if (savedFilterForNonBucketTypes && !ruleParams.customFilter) {
+          setRuleParams('customFilter', savedFilterForNonBucketTypes);


I'm finding that alerts aren't triggering if I don't have a filter set (Kibana sample data logs, logs_response_code_rates job, record type). Have you seen alerts with no filter?

There might be some odd results here as the job against the sample logs data had results going into the future. Running up 'now' looks better, but worth checking the query when no filter and the time range being applied.

Good catch!
It should be fixed with 61f41ce
What happened is that the query was missing 'LTE' filter

range: { timestamp: { gte: `now-${lookBackTimeInterval}`, lte: 'now', // previously missing }, },

It was working fine previously (probably by coincidence). I assume that after adding filters, Elasticsearch computes it differently, and future anomalies are detected. Adding the LTE filter fixes this behavior and makes it more predictable.

I'm also able to see alerts when no filter and the time range being applied (with proper lookback interval).

peteharverson · 2025-10-31T12:24:45Z

x-pack/platform/plugins/shared/ml/public/alerting/anomaly_detection_rule/anomaly_kql_filter.tsx

+          ) : (
+            <FormattedMessage
+              id="xpack.ml.anomalyDetectionAlert.anomalyFilterDescription"
+              defaultMessage="Filter which anomalies trigger this alert based on field values."


Looking at the labels used for some other rules (cluster health, index threshold), this would be more consistent...

Use a KQL expression to limit which anomalies trigger alerts.

Also I wonder if the filter query bar should be below the severity slider seeing as it's an optional field?

Updated the label in c8dd2a5
Also moved the query bar below the severity slider + moved the Test button closer to the interval input in f6d67f6

darnautov

Left some comments. I also think this change deserves some API integration tests.

darnautov · 2025-11-05T08:52:20Z

src/platform/packages/shared/response-ops/rule_params/anomaly_detection/v1.ts

  /** User's override for the top N buckets  */
  topNBuckets: schema.nullable(schema.number({ min: 1 })),
+  /** Optional KQL filter */
+  customFilter: schema.nullable(schema.string()),


I think we should validate query string on the schema level, because alerting rules can be created via the API.

Check if it's a valid KQL string

Should not contain a job_id. Perhaps it'd be better to have an allowlist of fields.

~~Added an allowlist along with schema validation in 803676d~~
As discussed, I added validation against disallow list: b1db8fb

darnautov · 2025-11-05T08:54:38Z

x-pack/platform/plugins/shared/ml/common/types/alerts.ts

  includeInterim: boolean;
  lookbackInterval: string | null | undefined;
  topNBuckets: number | null | undefined;
+  customFilter?: string | null;


Not sure about the name, because this filter and other params are not mutually exclusive. queryStringmaybe?

Renamed to kqlQueryString in 803676d

darnautov · 2025-11-05T08:56:26Z

x-pack/platform/plugins/shared/ml/common/util/alerting/detect_anomaly_alert_field_usage.ts

+export interface AnomalyAlertFieldUsage {
+  hasAnomalyScoreFilter: boolean;
+  hasInterimFilter: boolean;
+}


Would it be easier to validate the query string and disallow these fields?

And only show fields from allowlist in KQL bar suggestions

Totally makes sense to me, removed in c9fadcf

darnautov · 2025-11-05T08:59:11Z

x-pack/platform/plugins/shared/ml/public/alerting/anomaly_detection_rule/anomaly_kql_filter.tsx

+              true
+            );
+
+            setMlAnomaliesDataView(dataView);


potentially setting state for unmounted component here

Good catch, updated in fcc5584

darnautov · 2025-11-05T08:59:39Z

x-pack/platform/plugins/shared/ml/public/alerting/anomaly_detection_rule/anomaly_kql_filter.tsx

+        const fetchDataView = async () => {
+          try {
+            const allFields = await dataViewsService.getFieldsForWildcard({
+              pattern: '.ml-anomalies-*',


there is const for this index pattern

Updated in fcc5584

darnautov · 2025-11-05T09:01:05Z

x-pack/platform/plugins/shared/ml/public/alerting/anomaly_detection_rule/anomaly_kql_filter.tsx

+              return acc;
+            }, {});
+
+            const dataView = await dataViewsService.create(


you also need to clean up ad-hoc data views on unmount

Updated in fcc5584

darnautov · 2025-11-05T09:03:39Z

x-pack/platform/plugins/shared/ml/public/alerting/anomaly_detection_rule/anomaly_kql_filter.tsx

+    useEffect(
+      function parseAndNotifyFieldUsage() {
+        if (onFieldUsageChange) {
+          const usage = detectAnomalyAlertFieldUsage(value);
+          onFieldUsageChange(usage);
+        }
+      },
+      [value, onFieldUsageChange]
+    );


do we need this effect? can this logic be a part of onChange callback?

This part is gone within c9fadcf

darnautov · 2025-11-05T09:10:51Z

...atform/plugins/shared/ml/public/alerting/anomaly_detection_rule/ml_anomaly_alert_trigger.tsx

+  useEffect(
+    function handleFilterPreservationAcrossResultTypes() {


why do we need this effect? it is quite difficult to follow. is it here to restore the prev filter value when the user is clicking on different result type tiles?

Yup, I wanted to achieve such behavior:

The user types something into the query bar when the record type is record or influencer

They switch to the bucket result type -> the KQL bar becomes empty and disabled

They switch back to the record or influencer type -> the KQL bar is restored with the previous filter
When switching between record and influencer it is preserved.

peteharverson

Overall looks good. Tested each of the result types with a variety of filters. Just left one question on jobs without influencers.

peteharverson

Tested latest changes and LGTM

jgowdyelastic

Sorry, left a load of nit picks about the preferred use of ?? over ||
But otherwise, LGTM

x-pack/platform/plugins/shared/ml/public/alerting/anomaly_detection_rule/anomaly_kql_filter.tsx

...form/plugins/shared/ml/public/alerting/anomaly_detection_rule/get_relevant_anomaly_fields.ts

jgowdyelastic · 2025-11-13T13:18:53Z

...form/plugins/shared/ml/public/alerting/anomaly_detection_rule/get_relevant_anomaly_fields.ts

+
+  if (jobConfigs.length > 0) {
+    jobConfigs.forEach((job) => {
+      const detectors = job.analysis_config?.detectors || [];


nit

Suggested change

const detectors = job.analysis_config?.detectors || [];

const detectors = job.analysis_config?.detectors ?? [];

updated in 7ac1dee

Schema update for #240100

elasticmachine · 2025-11-27T13:39:41Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: c9f4eb9

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #86 / Search solution tests Search index details page Solution Nav - Search search index details page page loading error reload button shows details page again

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`ml`	2813	2815	+2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/ml-anomaly-utils`	13	14	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ml`	5.6MB	5.6MB	+37.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`ml`	86.4KB	86.7KB	+241.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/ml-anomaly-utils`	221	228	+7

ESLint disabled line counts

id	before	after	diff
`ml`	563	562	-1

Total ESLint disabled count

id	before	after	diff
`ml`	566	565	-1

History

💔 Build #364821 failed 6ca822c
💔 Build #361263 failed 7ac1dee
💔 Build #360564 failed 2eb304f
💔 Build #360407 failed d050da7
💛 Build #359080 was flaky 7d45666

cc @rbrtj

Resolves elastic#136392 This PR adds alert filtering capability for the Anomaly Detection Alerting Rule using the KQL bar. It provides proper auto-suggestions for the selected result and job type. How to test: ``` 1. Create a pre-configured Kibana sample e-commerce job without starting the datafeed. 2. Create an Anomaly Detection alerting rule based on the created job, e.g., with a 168h lookback interval and 168 latest buckets, select either the `record` or `influencer` result type. 3. Go to the AD jobs page and start the datafeed with `beginning of data` and `no end time` 4. Navigate back to the alerting rule, run the rule, and verify that it triggers an alert. 5. Create a new alerting rule with the desired filter. For example, if you alert was triggered for `customer_full_name.keyword: Eddie Lambert`, try filtering it out with: `NOT customer_full_name.keyword: "Eddie Lambert"` 6. Verify that the new rule does not trigger an alert for the specified `customer_full_name` 7. It's easier to use an email or slack connector to inspect the alerts. ``` If the user provides `record_score` or `influencer_score` in the KQL bar, the severity control is disabled and the provided query value is used. The same applies to the `is_interim` control. <img width="990" height="930" alt="image" src="https://github.com/user-attachments/assets/f53a4b8f-0191-46cd-8bca-913ec6077316" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…3236) Schema update for elastic#240100

Resolves elastic#136392 This PR adds alert filtering capability for the Anomaly Detection Alerting Rule using the KQL bar. It provides proper auto-suggestions for the selected result and job type. How to test: ``` 1. Create a pre-configured Kibana sample e-commerce job without starting the datafeed. 2. Create an Anomaly Detection alerting rule based on the created job, e.g., with a 168h lookback interval and 168 latest buckets, select either the `record` or `influencer` result type. 3. Go to the AD jobs page and start the datafeed with `beginning of data` and `no end time` 4. Navigate back to the alerting rule, run the rule, and verify that it triggers an alert. 5. Create a new alerting rule with the desired filter. For example, if you alert was triggered for `customer_full_name.keyword: Eddie Lambert`, try filtering it out with: `NOT customer_full_name.keyword: "Eddie Lambert"` 6. Verify that the new rule does not trigger an alert for the specified `customer_full_name` 7. It's easier to use an email or slack connector to inspect the alerts. ``` If the user provides `record_score` or `influencer_score` in the KQL bar, the severity control is disabled and the provided query value is used. The same applies to the `is_interim` control. <img width="990" height="930" alt="image" src="https://github.com/user-attachments/assets/f53a4b8f-0191-46cd-8bca-913ec6077316" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

… alerting rule (#241274) Follow up to #240100 Resolves #239946 This PR enables the `Create alert rule` action from the Anomalies Table, Single Metric Viewer chart and Anomaly Charts. It pre-populates the alert rule form with: **Alert settings**: - Job ID - Severity - `anomaly score - 5` - Result Type - Score **KQL Filter**: - Partition filter field (if present) - Over filter field (if present) - By filter field (if present) - Influencer filters - all influencers from the anomaly - Actual value threshold https://github.com/user-attachments/assets/26a1934b-191f-48dd-8882-d7a1e1d60534

… alerting rule (elastic#241274) Follow up to elastic#240100 Resolves elastic#239946 This PR enables the `Create alert rule` action from the Anomalies Table, Single Metric Viewer chart and Anomaly Charts. It pre-populates the alert rule form with: **Alert settings**: - Job ID - Severity - `anomaly score - 5` - Result Type - Score **KQL Filter**: - Partition filter field (if present) - Over filter field (if present) - By filter field (if present) - Influencer filters - all influencers from the anomaly - Actual value threshold https://github.com/user-attachments/assets/26a1934b-191f-48dd-8882-d7a1e1d60534

Enable alerts filtering with KQL bar

058996a

peteharverson changed the title ~~Enable alerts filtering with KQL bar~~ Oct 23, 2025

rbrtj added 2 commits October 27, 2025 11:34

Pass custom filters to rule execution logic

c8eed65

cleanup

cb34be0

rbrtj self-assigned this Oct 29, 2025

rbrtj added release_note:enhancement :ml backport:skip This PR does not require backporting Team:ML Team label for ML (also use :ml) t// v9.3.0 labels Oct 29, 2025

rbrtj requested a review from Copilot October 29, 2025 10:25

rbrtj marked this pull request as ready for review October 29, 2025 10:25

rbrtj requested review from a team as code owners October 29, 2025 10:25

rbrtj requested review from jgowdyelastic and peteharverson October 29, 2025 10:25

Merge branch 'main' into anomaly-detection-alerting-rule-filtering

6e06bd7

Copilot AI reviewed Oct 29, 2025

View reviewed changes

x-pack/platform/plugins/shared/ml/server/lib/alerts/alerting_service.ts Outdated Show resolved Hide resolved

...atform/plugins/shared/ml/public/alerting/anomaly_detection_rule/ml_anomaly_alert_trigger.tsx Show resolved Hide resolved

update snapshot

1e608e0

peteharverson reviewed Oct 31, 2025

View reviewed changes

rbrtj and others added 4 commits November 3, 2025 12:21

Add LTE filter and fix inverted is_interim logic

61f41ce

Merge branch 'main' into anomaly-detection-alerting-rule-filtering

2ee6b3b

Update label message

c8dd2a5

UI Adjustments

f6d67f6

rbrtj requested a review from peteharverson November 3, 2025 12:11

darnautov reviewed Nov 5, 2025

View reviewed changes

KQL Filter fields allowlist && rename KQl filter name to kqlQueryString

803676d

peteharverson reviewed Nov 6, 2025

View reviewed changes

Remove influencer field filters for jobs with no influencers

7d45666

rbrtj requested a review from peteharverson November 6, 2025 16:00

rbrtj added 3 commits November 12, 2025 09:27

Merge branch 'main' into anomaly-detection-alerting-rule-filtering

d050da7

Merge branch 'main' into anomaly-detection-alerting-rule-filtering

2eb304f

Merge branch 'main' into anomaly-detection-alerting-rule-filtering

a3789b9

peteharverson approved these changes Nov 12, 2025

View reviewed changes

jgowdyelastic approved these changes Nov 13, 2025

View reviewed changes

use nullish operator

7ac1dee

rbrtj mentioned this pull request Nov 17, 2025

[ML][Rules] Update Anomaly Detection alerting rule schema #243236

Merged

rbrtj added a commit that referenced this pull request Nov 21, 2025

[ML][Rules] Update Anomaly Detection alerting rule schema (#243236)

2bd3326

Schema update for #240100

rbrtj and others added 3 commits November 25, 2025 11:04

Merge branch 'main' into anomaly-detection-alerting-rule-filtering

6ca822c

remove duplicated export

4af87d3

Changes from node scripts/regenerate_moon_projects.js --update

bff386f

elastic-vault-github-plugin-prod bot requested review from a team as code owners November 25, 2025 10:42

jbudz approved these changes Nov 25, 2025

View reviewed changes

jloleysens approved these changes Nov 26, 2025

View reviewed changes

Merge branch 'main' into anomaly-detection-alerting-rule-filtering

c9f4eb9

darnautov approved these changes Nov 27, 2025

View reviewed changes

rbrtj merged commit 09509c8 into elastic:main Nov 27, 2025
12 checks passed

This was referenced Nov 28, 2025

[Internal][ML UI/AI Infra][Alerting]: Anomaly Detection: Alerting rule filtering elastic/docs-content#4145

Closed

[ML] Anomaly Detection: Adds an action to create an anomaly detection alerting rule #241274

Merged

eokoneyo pushed a commit to eokoneyo/kibana that referenced this pull request Dec 2, 2025

[ML][Rules] Update Anomaly Detection alerting rule schema (elastic#24…

27bcf1c

…3236) Schema update for elastic#240100

		useEffect(
		function handleFilterPreservationAcrossResultTypes() {

	const detectors = job.analysis_config?.detectors \|\| [];
	const detectors = job.analysis_config?.detectors ?? [];

Conversation

rbrtj commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticmachine commented Oct 29, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

github-actions bot commented Oct 29, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darnautov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rbrtj Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Labels

9 participants

rbrtj commented Oct 22, 2025 •

edited

Loading

rbrtj Nov 5, 2025 •

edited

Loading

elasticmachine commented Nov 27, 2025 •

edited

Loading