Adding linear retriever to support weighted sums of sub-retrievers

[pmpailis]

This PR adds a new linear retriever to facilitate hybrid search, that would be able to linearly combine the results of other sub-retrievers and compute the final score of a document based on the weighted sum of each sub-components.

Each sub-component can specify the following elements:

• retriever -> specifies how we will compute the top documents
• normalizer -> specifies how we want to normalize the top documents for this retriever (so that we can ensure that all scores fall within the same range)
• weight -> the weight for the normalized score if the final weighted sum computation

Pagination is similar to that of rrf's retriever, i.e. we compute the global rank_window_size docs and pagination is only available within these bounds.

So, working through an example, let's say that we perform a hybrid search query where:

• we want to run a simple string query through a standard retriever, and normalize the scores to a [0, 1] range
• we want to run knn search through the knn retriever, without normalizing the documents as well
• compute the final score for the retriever as score = 1.5 * standard + 2.5 * knn

Sample syntax:

GET /retrievers_example/_search
{
    "retriever": {
        "linear": {
            "retrievers": [
                {
                        "retriever": {
                            "standard": {
                                "query": {
                                    "simple_query_string": {
                                        "query": "artifical intelligence in medicine",
                                        "fields": [
                                            "text"
                                        ]
                                    }
                                }
                            }
                        },
                        "weight": 1.5,
                        "normalizer": "minmax"
                },
                {
                        "retriever": {
                            "knn": {
                                "field": "vector",
                                "query_vector": [
                                    0.23,
                                    0.67,
                                    0.89
                                ],
                                "k": 3,
                                "num_candidates": 5
                            }
                        },
                        "weight": 2.5
                }
            ],
            "rank_window_size": 10
        }
    }
}

[github-actions]

Documentation preview:

• ✨ Changed pages

[elasticsearchmachine]

Hi @pmpailis, I've created a changelog YAML for you.

[benwtrent]

Looking much better. I have a concern around testing:

Do we have a test that specifically exercises the path when the different retrievers return different doc IDs? (e.g. they match non-overlapping doc sets).

[pmpailis]

Do we have a test that specifically exercises the path when the different retrievers return different doc IDs? (e.g. they match non-overlapping doc sets).

Added a test to account for this in ea1787f

[benwtrent]

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 120222