[ResponseOps] Automatically copy source data into the alerts-as-data documents for other ES Query rule types by doakalexi · Pull Request #230010 · elastic/kibana

doakalexi · 2025-07-30T21:30:59Z

Resolves #190947

Summary

This PR updates the all the es query rule types to automatically copy source data into the alerts-as-data documents, instead of asking the customer to select fields in the UI. This PR also removes the option to select source fields in the rule form UI.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

To verify

I liked to use the sample data to test this. You can add it on the homepage ( The sample data logs will have a couple fields that will get written to the doc like message)
Create ES Query KQL and DSL rules.
Go to dev tools and run the query below to verify that the query results that are also ecs fields are written to the alert doc.

GET .internal.alerts-stack.alerts-default*/_search

…xi/kibana into update/copy-es-query-source-data

elasticmachine · 2025-08-14T17:50:00Z

Pinging @elastic/response-ops (Team:ResponseOps)

dominiqueclarke · 2025-08-21T13:30:52Z

...y/test/api_integration_deployment_agnostic/apis/alerting/es_query/query_dsl_with_group_by.ts


-        expect(resp.hits.hits[0]._source).property('host.name', 'host-0');
+        expect(resp.hits.hits[0]._source).property('host.name');
+        expect(get(resp.hits.hits[0]._source, 'host.name')).eql(['host-0', 'host-0', 'host-0']);


Why it an array of the 3 of the same value here when previously it was the single host-0?

They aren't deduped anymore bc we aren't doing an aggregation for these fields. We are just parsing them out of the hits similar to how to this works for the ES|QL query type

This seems a little problematic. At least for this field, I'd think we'd want to collapse to a single string, since host.name is considered a non-array in ECS. But in the case where the values are NOT the same, what would we do? Pick the first one? I'm guessing this could get more complicated for other fields.

I know we suggest folks access the .alerts documents via fields, I think partially because we sometimes write out the full field name instead of using nested objects. But it also means that technically having a three element host.name array value shouldn't cause any kind of issue.

But it does seem odd.

One thing I'm wondering is if it would make sense to throw these values in a Set to remove dups - can we do that with all the fields? I guess probably just fields that aren't already ECS array values (since they may expect dups). Then we'd only see the multi-element arrays when there aren't dups.

Yeah that's fair! We can do a set to remove dupes

We had a discussion on this when I first made the change to write ECS fields to the alert doc for ES|QL

@mikecote thoughts? I guess the problem with de-duping (see discussion link in previous comment) is going to be that the values won't "line up" anymore, if some fields have dup'd values vs not. But I'm not sure there's much value there anyway, not clear that the values would end up "lining up" after making it into ES and then out again (through Kibana).

I'm more concerned that the "source" version won't validate against an ECS-based mapping anymore. And that there's probably an expectation that "fields" which aren't supposed to be arrays, actually have multiple elements. But I guess we don't really support "source" for these anyway?

I guess I'm kind of thinking that the current code - not deduping - is probably the easiest thing for people to understand. And I guess it makes sense if folks are filtering on these fields.

Yeah it appears the idea of not de-dupping was to have the same count of values in all the arrays so they line up. I don't have a strong opinion on whether we should de-dup or not, given it will still function for it's main purpose which is conditional actions and maintenance windows.

I recall communicating the oddity of arrays to stakeholders when we first did this and it didn't appear to be a concern for anyone at that time.

Ya, I think I'm fine with not de-dup'ing then. I'm worried that we'll run into problems like this one: #221252 (comment) , where somewhere in our code we assume an arity that is wrong when we run.

…query-source-data

…xi/kibana into update/copy-es-query-source-data

doakalexi · 2025-09-08T20:04:15Z

Tried this on a DSL rule (so far) - I ran a rule against the event log, as I was interested to see how many of the fields would make it over.

These all seem like fields we should NOT be copying over, though maybe it's ok? And even if it's ok, it's likely going to be strange for event.start to have multiple different dates in there. If someone is searching on them, they may be surprised to see a match. event.category is weird also - I think that should be a flat array and not array of arrays.

So feels like we will need to do some coersion for fields that are arrays, at a minimum.

list of "arrayed" fields in the alert

Resolved in this commit, a324cdb

ymao1 · 2025-09-10T19:11:40Z

If I set the size parameter to 0, I get an execution error

[2025-09-10T15:09:43.966-04:00][ERROR][plugins.alerting.es-query] Executing Rule default:.es-query:36e24b02-4727-422f-a727-0c2684f55886 has resulted in Error: Cannot read properties of undefined (reading 'hits') - TypeError: Cannot read properties of undefined (reading 'hits')
    at forEach (parse_aggregation_results.ts:118:32)
    at Array.forEach (<anonymous>)
    at parseAggregationResults (parse_aggregation_results.ts:116:26)
    at fetchEsQuery (fetch_es_query.ts:156:43)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at executor (executor.ts:120:7)
    at Object.executor (rule_type.ts:227:14)
    at rule_type_runner.ts:214:28
    at TaskRunnerTimer.runWithTimer (task_runner_timer.ts:50:20)
    at RuleTypeRunner.run (rule_type_runner.ts:189:57)
    at TaskRunner.runRule (task_runner.ts:373:9)
    at TaskRunner.run (task_runner.ts:695:9)
    at TaskManagerRunner.run (task_runner.ts:419:22)

so I guess it is dependent on the query returning hits to copy over in the source fields to work correctly?

doakalexi · 2025-09-10T19:34:17Z

If I set the size parameter to 0, I get an execution error

[2025-09-10T15:09:43.966-04:00][ERROR][plugins.alerting.es-query] Executing Rule default:.es-query:36e24b02-4727-422f-a727-0c2684f55886 has resulted in Error: Cannot read properties of undefined (reading 'hits') - TypeError: Cannot read properties of undefined (reading 'hits')
    at forEach (parse_aggregation_results.ts:118:32)
    at Array.forEach (<anonymous>)
    at parseAggregationResults (parse_aggregation_results.ts:116:26)
    at fetchEsQuery (fetch_es_query.ts:156:43)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at executor (executor.ts:120:7)
    at Object.executor (rule_type.ts:227:14)
    at rule_type_runner.ts:214:28
    at TaskRunnerTimer.runWithTimer (task_runner_timer.ts:50:20)
    at RuleTypeRunner.run (rule_type_runner.ts:189:57)
    at TaskRunner.runRule (task_runner.ts:373:9)
    at TaskRunner.run (task_runner.ts:695:9)
    at TaskManagerRunner.run (task_runner.ts:419:22)

so I guess it is dependent on the query returning hits to copy over in the source fields to work correctly?

Ah sorry, I will fix this. Yeah we won't copy over data if there aren't any hits

Resolved in this commit, b47b3c1

github-actions · 2025-09-17T20:36:37Z

@doakalexi, it looks like you're updating the parameters for a rule type!

Please review the guidelines for making additive changes to rule type parameters and determine if your changes require an intermediate release.

ymao1

LGTM! Verified works as described and updating a rule with a previously set source fields param continues to work. The copying of the source fields into an array with possible repeated values does seem a little funky but it looks like it's working the same way as ES|QL rules and we haven't gotten any complaints from that department so seems fine.

I think we need a doc issue to add this behavior to the docs and to specify that the size field needs to be > 0 in order for any source fields to get copied.

pmuellr

LGTM, that getSourceFields() test is ... interesting! :-) We'll see how it pans out over time.

pmuellr · 2025-09-23T15:11:40Z

x-pack/platform/plugins/shared/stack_alerts/server/rule_types/es_query/util.test.ts

+    it('should generate the correct source fields', async () => {
+      const sourceFields = getSourceFields();
+      expect(sourceFields).toMatchInlineSnapshot(`
+        Array [


Oh wow. I wonder how well this will work over time? :-)

Thanks for reviewing! Do you think I should change anything? I kinda just did a check of the fields

Nah, I'm interested to see how this plays out. If it's causes too much churn, I think we can just remove it ... or maybe someone would want the snapshot stored in an external file or something.

We'll see!

elasticmachine · 2025-09-23T23:29:58Z

💚 Build Succeeded

Buildkite Build
Commit: 7cead99

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`stackAlerts`	297	296	-1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`triggersActionsUi`	493	492	-1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`stackAlerts`	65.0KB	63.5KB	-1.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`stackAlerts`	28.0KB	27.5KB	-438.0B
`triggersActionsUi`	104.8KB	104.7KB	-107.0B
total			-545.0B

Unknown metric groups

API count

id	before	after	diff
`triggersActionsUi`	497	496	-1

ESLint disabled line counts

id	before	after	diff
`stackAlerts`	25	24	-1

Total ESLint disabled count

id	before	after	diff
`stackAlerts`	25	24	-1

History

💔 Build #342217 failed 5b60d00
💔 Build #342098 failed de337d6
💛 Build #340896 was flaky 5db4197
💔 Build #340559 failed 8632e82
💛 Build #339518 was flaky 96c9e17

dominiqueclarke

LGTM

…documents for other ES Query rule types (elastic#230010) Resolves elastic#190947 ## Summary This PR updates the all the es query rule types to automatically copy source data into the alerts-as-data documents, instead of asking the customer to select fields in the UI. This PR also removes the option to select source fields in the rule form UI. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - I liked to use the sample data to test this. You can add it on the homepage ( The sample data logs will have a couple fields that will get written to the doc like message) - Create ES Query KQL and DSL rules. - Go to [dev tools](http://localhost:5601/app/dev_tools#/console) and run the query below to verify that the query results that are also ecs fields are written to the alert doc. ``` GET .internal.alerts-stack.alerts-default*/_search ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…documents for other ES Query rule types (#230010) Resolves #190947 ## Summary This PR updates the all the es query rule types to automatically copy source data into the alerts-as-data documents, instead of asking the customer to select fields in the UI. This PR also removes the option to select source fields in the rule form UI. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - I liked to use the sample data to test this. You can add it on the homepage ( The sample data logs will have a couple fields that will get written to the doc like message) - Create ES Query KQL and DSL rules. - Go to [dev tools](http://localhost:5601/app/dev_tools#/console) and run the query below to verify that the query results that are also ecs fields are written to the alert doc. ``` GET .internal.alerts-stack.alerts-default*/_search ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

… in the current alert (#235854) Follow on from #190947 ## Summary We noticed while testing this [PR](#230010), that we don't clear out source data when there are no hits returned in the next rule execution. This can happen when a user edits the ES Query rule size to be 0. This PR adds a new function to overwrite fields that aren't in the new payload with empty arrays if they were populated in the existing alert. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify 1. Use sample data or data that contains ECS fields 2. Create a KQL of DSL ES Query rule 3. Let the rule run and verify that the source data is populated in the alert doc. You can run this query below: ``` GET /.alerts-stack.alerts-default/_search ``` 4. Edit the rule and set the size to 0 5. Let the rule run again and verify that the source fields are set to empty arrays. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…documents for other ES Query rule types (elastic#230010) Resolves elastic#190947 ## Summary This PR updates the all the es query rule types to automatically copy source data into the alerts-as-data documents, instead of asking the customer to select fields in the UI. This PR also removes the option to select source fields in the rule form UI. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - I liked to use the sample data to test this. You can add it on the homepage ( The sample data logs will have a couple fields that will get written to the doc like message) - Create ES Query KQL and DSL rules. - Go to [dev tools](http://localhost:5601/app/dev_tools#/console) and run the query below to verify that the query results that are also ecs fields are written to the alert doc. ``` GET .internal.alerts-stack.alerts-default*/_search ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

… in the current alert (elastic#235854) Follow on from elastic#190947 ## Summary We noticed while testing this [PR](elastic#230010), that we don't clear out source data when there are no hits returned in the next rule execution. This can happen when a user edits the ES Query rule size to be 0. This PR adds a new function to overwrite fields that aren't in the new payload with empty arrays if they were populated in the existing alert. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify 1. Use sample data or data that contains ECS fields 2. Create a KQL of DSL ES Query rule 3. Let the rule run and verify that the source data is populated in the alert doc. You can run this query below: ``` GET /.alerts-stack.alerts-default/_search ``` 4. Edit the rule and set the size to 0 5. Let the rule run again and verify that the source fields are set to empty arrays. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

doakalexi and others added 11 commits July 30, 2025 14:30

Updating source fields for es query rule types

f9a27a2

Fixing types checks

c890687

Merge branch 'main' into update/copy-es-query-source-data

cde84bf

Fixing test failures

8b917e1

Merge branch 'update/copy-es-query-source-data' of github.com:doakale…

f828b23

…xi/kibana into update/copy-es-query-source-data

Fixing translations

dee2adb

Fixing issues with source fields

b47b5ec

[CI] Auto-commit changed files from 'security: 3rd-party dependencies'

3ffbb38

Moving around source field function

5a3b11c

Merge branch 'update/copy-es-query-source-data' of github.com:doakale…

1c4c454

…xi/kibana into update/copy-es-query-source-data

[CI] Auto-commit changed files from 'security: 3rd-party dependencies'

6280168

doakalexi added Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// release_note:enhancement backport:skip This PR does not require backporting v9.2.0 labels Aug 5, 2025

doakalexi changed the title ~~Updating source fields for es query rule types~~ Aug 5, 2025

doakalexi and others added 6 commits August 7, 2025 11:30

Fixing test changes

c1ff894

Merge branch 'update/copy-es-query-source-data' of github.com:doakale…

aa5a4ce

…xi/kibana into update/copy-es-query-source-data

Fixing rule tests

b919a32

Fixing test

b4311c2

Merge branch 'main' into update/copy-es-query-source-data

df5c696

Merge branch 'main' into update/copy-es-query-source-data

fa38849

doakalexi marked this pull request as ready for review August 14, 2025 17:49

doakalexi requested review from a team as code owners August 14, 2025 17:49

Merge branch 'main' into update/copy-es-query-source-data

6732468

dominiqueclarke reviewed Aug 21, 2025

View reviewed changes

Merge branch 'main' of github.com:elastic/kibana into update/copy-es-…

eca13ba

…query-source-data

darnautov requested a review from dominiqueclarke August 27, 2025 12:13

doakalexi added 2 commits September 8, 2025 13:01

addressing pr comments

a324cdb

Merge branch 'update/copy-es-query-source-data' of github.com:doakale…

745c61c

…xi/kibana into update/copy-es-query-source-data

doakalexi added 3 commits September 8, 2025 13:40

Fixing type check

cfc2499

Fixing type checks again

ae47007

Updating OAS docs

97cfd2a

doakalexi and others added 4 commits September 10, 2025 13:04

Fixing when no hits are returned

b47b3c1

Merge branch 'main' into update/copy-es-query-source-data

22e0710

Merge branch 'main' into update/copy-es-query-source-data

96c9e17

Merge branch 'main' into update/copy-es-query-source-data

8632e82

Merge branch 'main' into update/copy-es-query-source-data

5db4197

ymao1 approved these changes Sep 19, 2025

View reviewed changes

Merge branch 'main' into update/copy-es-query-source-data

de337d6

doakalexi mentioned this pull request Sep 22, 2025

[ResponseOps] Overwrite array fields with empty array if not returned in the current alert #235854

Merged

1 task

Merge branch 'main' into update/copy-es-query-source-data

5b60d00

pmuellr approved these changes Sep 23, 2025

View reviewed changes

Merge branch 'main' into update/copy-es-query-source-data

7cead99

dominiqueclarke approved these changes Sep 24, 2025

View reviewed changes

doakalexi merged commit a0bcc4e into elastic:main Sep 24, 2025
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ResponseOps] Automatically copy source data into the alerts-as-data documents for other ES Query rule types#230010

[ResponseOps] Automatically copy source data into the alerts-as-data documents for other ES Query rule types#230010
doakalexi merged 34 commits intoelastic:mainfrom
doakalexi:update/copy-es-query-source-data

doakalexi commented Jul 30, 2025 •

edited

Loading

elasticmachine commented Aug 14, 2025

dominiqueclarke Aug 21, 2025

doakalexi Aug 21, 2025

pmuellr Sep 2, 2025

doakalexi Sep 2, 2025

doakalexi Sep 2, 2025 •

edited

Loading

pmuellr Sep 3, 2025

mikecote Sep 4, 2025

pmuellr Sep 4, 2025

doakalexi commented Sep 8, 2025

ymao1 commented Sep 10, 2025

doakalexi commented Sep 10, 2025 •

edited

Loading

github-actions bot commented Sep 17, 2025

ymao1 left a comment

pmuellr left a comment

pmuellr Sep 23, 2025

doakalexi Sep 23, 2025 •

edited

Loading

pmuellr Sep 25, 2025

elasticmachine commented Sep 23, 2025

API count

ESLint disabled line counts

Total ESLint disabled count

dominiqueclarke left a comment

Uh oh!

Labels

8 participants

Conversation

doakalexi commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

To verify

elasticmachine commented Aug 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doakalexi Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doakalexi commented Sep 8, 2025

ymao1 commented Sep 10, 2025

doakalexi commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Sep 17, 2025

ymao1 left a comment

Choose a reason for hiding this comment

pmuellr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doakalexi Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 23, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

dominiqueclarke left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

8 participants

doakalexi commented Jul 30, 2025 •

edited

Loading

doakalexi Sep 2, 2025 •

edited

Loading

doakalexi commented Sep 10, 2025 •

edited

Loading

doakalexi Sep 23, 2025 •

edited

Loading