[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights#244109
Merged
rbrtj merged 6 commits intoelastic:mainfrom Dec 16, 2025
Merged
Conversation
Contributor
Author
|
/ci |
walterra
reviewed
Dec 3, 2025
| category_counts: { | ||
| filters: { | ||
| filters: { | ||
| 'runTask ended no files to process': { |
Contributor
There was a problem hiding this comment.
I think it would be better if this was a generated name like filter_0001 or similar. In x-pack/platform/packages/shared/ml/aiops_log_rate_analysis/queries/fetch_significant_term_p_values.ts we do something similar where we name aggs like aggs[distinct_count_${index}] = ... and aggs[sig_term_p_value_${index}] = ....
walterra
reviewed
Dec 3, 2025
|
|
||
| try { | ||
| mSearchresponse = await esClient.msearch({ searches }, { signal: abortSignal, maxRetries: 0 }); | ||
| response = await esClient.search(request, { signal: abortSignal, maxRetries: 0 }); |
Contributor
There was a problem hiding this comment.
Because we raised the category limit to 1000 again, it means this can be a query with up to 1000 filter aggs. We discussed also adding an inner async queue here to split this up into multiple search with each up to 100 filters for example.
Contributor
Author
There was a problem hiding this comment.
As discussed offline, a single query with up to 1000 filters should be fine.
rbrtj
added
release_note:fix
:ml
Team:ML
Contributor
|
Pinging @elastic/ml-ui (:ml) |
walterra
approved these changes
Dec 15, 2025
Contributor
walterra
left a comment
walterra
left a comment
There was a problem hiding this comment.
Latest code changes LGTM.
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
History
cc @rbrtj |
peteharverson
approved these changes
Dec 15, 2025
Contributor
peteharverson
left a comment
peteharverson
left a comment
There was a problem hiding this comment.
LGTM. Tested log rate analysis in the ML app and in the custom threshold alert details page to verify there were no regressions.
Contributor
|
Starting backport for target branches: 8.19, 9.1, 9.2 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 16, 2025
… contextual insights (elastic#244109) ## Summary Resolves elastic#235562 - Moves multiple `msearch` requests to a single search request using a filters agg, making it more efficient. - Introduces support for random sampling in category count queries. - Enables text field analysis in alerts and contextual insights. Before: <img width="1065" height="893" alt="image" src="https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146" /> After: <img width="1023" height="799" alt="image" src="https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107" /> (cherry picked from commit 651d73a)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 16, 2025
… contextual insights (elastic#244109) ## Summary Resolves elastic#235562 - Moves multiple `msearch` requests to a single search request using a filters agg, making it more efficient. - Introduces support for random sampling in category count queries. - Enables text field analysis in alerts and contextual insights. Before: <img width="1065" height="893" alt="image" src="https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146" /> After: <img width="1023" height="799" alt="image" src="https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107" /> (cherry picked from commit 651d73a)
Contributor
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
rbrtj
added a commit
to rbrtj/kibana
that referenced
this pull request
Dec 16, 2025
… contextual insights (elastic#244109) ## Summary Resolves elastic#235562 - Moves multiple `msearch` requests to a single search request using a filters agg, making it more efficient. - Introduces support for random sampling in category count queries. - Enables text field analysis in alerts and contextual insights. Before: <img width="1065" height="893" alt="image" src="https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146" /> After: <img width="1023" height="799" alt="image" src="https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107" /> (cherry picked from commit 651d73a) # Conflicts: # x-pack/platform/packages/shared/ml/aiops_log_rate_analysis/queries/fetch_categories.test.ts # x-pack/platform/packages/shared/ml/aiops_log_rate_analysis/queries/fetch_category_counts.test.ts # x-pack/platform/packages/shared/ml/aiops_log_rate_analysis/queries/fetch_category_counts.ts
kibanamachine
added a commit
that referenced
this pull request
Dec 16, 2025
…sis in contextual insights (#244109) (#246497) # Backport This will backport the following commits from `main` to `9.2`: - [[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)](#244109) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Robert Jaszczurek","email":"92210485+rbrtj@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-12-16T08:33:02Z","message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Team:ML","backport:version","v9.3.0","ci:beta-faster-pr-build","v9.2.3","v9.1.9","v8.19.9"],"title":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights","number":244109,"url":"https://github.com/elastic/kibana/pull/244109","mergeCommit":{"message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0"}},"sourceBranch":"main","suggestedTargetBranches":["9.2","9.1","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/244109","number":244109,"mergeCommit":{"message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0"}},{"branch":"9.2","label":"v9.2.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Robert Jaszczurek <92210485+rbrtj@users.noreply.github.com>
kibanamachine
added a commit
that referenced
this pull request
Dec 16, 2025
…sis in contextual insights (#244109) (#246496) # Backport This will backport the following commits from `main` to `9.1`: - [[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)](#244109) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Robert Jaszczurek","email":"92210485+rbrtj@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-12-16T08:33:02Z","message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Team:ML","backport:version","v9.3.0","ci:beta-faster-pr-build","v9.2.3","v9.1.9","v8.19.9"],"title":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights","number":244109,"url":"https://github.com/elastic/kibana/pull/244109","mergeCommit":{"message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0"}},"sourceBranch":"main","suggestedTargetBranches":["9.2","9.1","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/244109","number":244109,"mergeCommit":{"message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0"}},{"branch":"9.2","label":"v9.2.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Robert Jaszczurek <92210485+rbrtj@users.noreply.github.com>
rbrtj
added a commit
that referenced
this pull request
Dec 16, 2025
…ysis in contextual insights (#244109) (#246509) # Backport This will backport the following commits from `main` to `8.19`: - [[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)](#244109) <!--- Backport version: 10.2.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Robert Jaszczurek","email":"92210485+rbrtj@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-12-16T08:33:02Z","message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Team:ML","backport:version","v9.3.0","ci:beta-faster-pr-build","v9.2.3","v9.1.9","v8.19.9"],"title":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights","number":244109,"url":"https://github.com/elastic/kibana/pull/244109","mergeCommit":{"message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/244109","number":244109,"mergeCommit":{"message":"[ML] Log Rate Analysis: Optimizes & Re-enables text field analysis in contextual insights (#244109)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/235562\n\n- Moves multiple `msearch` requests to a single search request using a\nfilters agg, making it more efficient.\n- Introduces support for random sampling in category count queries.\n- Enables text field analysis in alerts and contextual insights.\n\nBefore:\n<img width=\"1065\" height=\"893\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/a15bf9b9-4d1a-4bcc-922b-4a6f0f282146\"\n/>\n\nAfter:\n<img width=\"1023\" height=\"799\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/55bc46aa-ab91-4867-b5dc-7d7692a62107\"\n/>","sha":"651d73a5fe6d4c04e78f1b2ddaa03c38f3308ef0"}},{"branch":"9.2","label":"v9.2.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/246497","number":246497,"state":"OPEN"},{"branch":"9.1","label":"v9.1.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/246496","number":246496,"state":"OPEN"},{"branch":"8.19","label":"v8.19.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #235562
msearchrequests to a single search request using a filters agg, making it more efficient.Before:
After:
