Use msearch to fetch the alerts for maintenance windows with scoped query

[ersin-erdal]

Currently we use a search with a top_hits aggregation to get the alerts filtered with the scoped queries of the active maintenance windows. But since top_hits aggregation has a fixed limit of 100 hits we cannot use our max alerts limit to get all the alerts an execution can generate.

This PR replaces search with msearch and removes the aggregation from the search query so we can use the maxAlerts limit for the response size.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

Previous query with top hits aggregation:

{
    "04e5b9d5-fa0b-4a3b-86b5-2ae8acdaf002": {
        "filter": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "bool": {
                            "should": [
                                {
                                    "match_phrase": {
                                        "kibana.alert.rule.rule_type_id": "test-rule"
                                    }
                                }
                            ],
                            "minimum_should_match": 1
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.execution.uuid": "99e07957-fe8e-46f0-a603-e8639807e334"
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.uuid": "cea45fdb-00ef-4f70-9199-477414d7aee4"
                        }
                    },
                    {
                        "term": {
                            "event.action": "open"
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        },
        "aggs": {
            "alertId": {
                "top_hits": {
                    "size": 100,
                    "_source": {
                        "includes": [
                            "kibana.alert.uuid"
                        ]
                    }
                }
            }
        }
    },
    "7b1b1ce0-4e34-48e7-a8f9-e320e1b52f95": {
        "filter": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "bool": {
                            "should": [
                                {
                                    "match_phrase": {
                                        "kibana.alert.rule.name": "test"
                                    }
                                }
                            ],
                            "minimum_should_match": 1
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.execution.uuid": "99e07957-fe8e-46f0-a603-e8639807e334"
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.uuid": "cea45fdb-00ef-4f70-9199-477414d7aee4"
                        }
                    },
                    {
                        "term": {
                            "event.action": "open"
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        },
        "aggs": {
            "alertId": {
                "top_hits": {
                    "size": 100,
                    "_source": {
                        "includes": [
                            "kibana.alert.uuid"
                        ]
                    }
                }
            }
        }
    }
}

New msearch:

[
    {},
    {
        "query": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "bool": {
                            "should": [
                                {
                                    "match_phrase": {
                                        "kibana.alert.rule.rule_type_id": "test-rule"
                                    }
                                }
                            ],
                            "minimum_should_match": 1
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.execution.uuid": "ef1a249a-cfab-4195-82e4-a0ce3d73ed4c"
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.uuid": "cea45fdb-00ef-4f70-9199-477414d7aee4"
                        }
                    },
                    {
                        "term": {
                            "event.action": "open"
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        },
        "runtime_mappings": {
            "runtime_maintenance_window_id": {
                "script": {
                    "source": "emit('04e5b9d5-fa0b-4a3b-86b5-2ae8acdaf002');"
                },
                "type": "keyword"
            }
        },
        "fields": [
            "kibana.alert.uuid",
            "runtime_maintenance_window_id"
        ],
        "_source": false,
        "size": 1000,
        "track_total_hits": true
    },
    {},
    {
        "query": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "bool": {
                            "should": [
                                {
                                    "match_phrase": {
                                        "kibana.alert.rule.name": "test"
                                    }
                                }
                            ],
                            "minimum_should_match": 1
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.execution.uuid": "ef1a249a-cfab-4195-82e4-a0ce3d73ed4c"
                        }
                    },
                    {
                        "term": {
                            "kibana.alert.rule.uuid": "cea45fdb-00ef-4f70-9199-477414d7aee4"
                        }
                    },
                    {
                        "term": {
                            "event.action": "open"
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        },
        "runtime_mappings": {
            "runtime_maintenance_window_id": {
                "script": {
                    "source": "emit('7b1b1ce0-4e34-48e7-a8f9-e320e1b52f95');"
                },
                "type": "keyword"
            }
        },
        "fields": [
            "kibana.alert.uuid",
            "runtime_maintenance_window_id"
        ],
        "_source": false,
        "size": 1000,
        "track_total_hits": true
    }
]

[ymao1]

Changes LGTM. Wondering if we have a functional test for MW with scoped queries already? If not, maybe we should add one.

[ymao1]

nit: should we use the optional accessor instead? Using ! will throw an error and cause the whole thing to fail if we can't access the field

[ersin-erdal]

I used! because we add this field in order to return the related mw. it should be impossible not to access it.
But i will modify it and skip the hit if the field doesn't exist.

For the functional test: Yes we already have a functional test. It didn't fail before because we set the maxAlerts limit to 20 in the test configs whereas it was 1000 for the real runtime. Now I set it to 110 in order to be sure that we exceed the fixed limit of 100 in ES.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 07887ed

Failed CI Steps

• FTR Configs #32
• FTR Configs #61
• FTR Configs #90
• FTR Configs #109

Metrics [docs]

✅ unchanged

History

• 💛 Build #303199 was flaky 94031a8
• 💔 Build #303113 failed ec3d879
• 💔 Build #303014 failed 7c726e7
• 💔 Build #302898 failed f0b3d55
• 💛 Build #302856 was flaky 60438a4

[kibanamachine]

Starting backport for target branches: 8.18, 8.19, 9.0

https://github.com/elastic/kibana/actions/runs/15356842953

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 221702

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 221702 locally
cc: @ersin-erdal

[ersin-erdal]

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.19
✅	8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation