Support querying multiple indices with the simplified linear retriever

[ioanatia]

This only handles the linear retriever, but the same approach can be used for the RRF retriever.
I just haven't removed the restriction for RRF yet, we can do it in a follow up.
It should be as simple as removing the check we have for RRF and adding tests.

And once we do this change for both retrievers, we can adjust the docs too.

When we rewrite the linear retriever, we form two sub-retrievers:

linear
   standard retriever with lexical query, normalizer:<specified_normalizer>
   linear, normalizer:<specified_normalizer>
      standard retriever normalizer:<specified_normalizer>
          match on semantic_text field 1,
      standard retriever normalizer:<specified_normalizer>
          match on semantic_text field 2
      ...

The lexical query uses multi match at the moment.
For the semantic part, we group by (inferenceId, field_name) and issue a match query for each group.
For each group, if the semantic_text field does not exist in all queried indices, we add an additional filter on the index names.

Some examples:

Let's assume we have index A and index B and both have the same semantic text fields using the same inference IDs: semantic_field_1 and semantic_field_2.
And that we issue a query to query the lexical_field_*, semantic_field_1, semantic_field_2.

The linear retriever will be rewritten to:

linear
  // Lexical group
  standard retriever using multi-match on `lexical_field_*` normalizer:minmax
    multi
  // Semantic group
  linear normalizer:minmax
    standard { match on semantic_field_1}, weight:1 normalizer:minmax
    standard { match on semantic_field_2}, weight:1 normalizer:minmax

Let's now assume that indexB has an extra semantic_field_3 and that we query:
["lexical_field_*^3", "semantic_field_1^10", "semantic_field_2^20", "semantic_field_3^30"].

The linear retriever will be rewritten to:

linear
  // Lexical group
  standard retriever `lexical_field_*` normalizer:minmax
   bool:
      should:
         multi_match ["lexical_field_*^3", "semantic_field_3^30"] filter on indexA 
         multi_match ["lexical_field_*^3"] filter on indexB
  // Semantic group
  linear normalizer:minmax
    standard { match on semantic_field_1}, weight:1 normalizer:minmax
    standard { match on semantic_field_2}, weight:1 normalizer:minmax
    standard { match on semantic_field_3, filter on indexB}, weight:1 normalizer:minmax

If only semantic_text field is queried, we don't need to have 2 levels of normalization, so we rewrite to:

linear
  standard retriever for lexical query, normalizer:minmax
  standard retriever, normalizer:minmax
     match on semantic_text fieldd

[elasticsearchmachine]

Hi @ioanatia, I've created a changelog YAML for you.

[ioanatia]

we could be in this situation when:

1. No query fields are provided and we need to use the ones from index.query.default_field index setting and not all indices have the same value for this setting, resulting in different list of fields that need to be queried per index. This case is uncommon.
2. One of the fields from the list that is provided to the linear retriever is a semantic_text field that is not present in all indices. This will be more common. It could be that a field with the same name does not even exist in all the other indices, or that it was mapped to another mapping type. We don't really know since we don't have access to the mappings.

From what I tested, the current approach where compose a boolean query and we filter by index names, seems to mitigate both cases.
If we find it too complicated, I can change this to raise an exception for both of these cases, so that we always use a single multi match query.

[Mikep86]

One of the fields from the list that is provided to the linear retriever is a semantic_text field that is not present in all indices. This will be more common. It could be that a field with the same name does not even exist in all the other indices, or that it was mapped to another mapping type. We don't really know since we don't have access to the mappings.

Agreed this will be more common, enough so to the point that IMO we need to handle it. I think it's also worth pointing out that we can be in this case when differentNonInferenceFields == true and lexicalQueryBuilders is not empty.

[Mikep86]

Nice work! I reviewed just the core logic for now, not the tests. I have some suggestions about how we may be able to simplify and handle more edge cases at the same time.

[Mikep86]

One of the fields from the list that is provided to the linear retriever is a semantic_text field that is not present in all indices. This will be more common. It could be that a field with the same name does not even exist in all the other indices, or that it was mapped to another mapping type. We don't really know since we don't have access to the mappings.

Agreed this will be more common, enough so to the point that IMO we need to handle it. I think it's also worth pointing out that we can be in this case when differentNonInferenceFields == true and lexicalQueryBuilders is not empty.

[elasticsearchmachine]

Pinging @elastic/search-relevance (Team:Search - Relevance)

[ioanatia]

the reason why I went through these hoops for the lexical query is because we are comparing boolean queries.
and AFAICS I can't guarantee the order of the should clauses which are held in a List<QueryBuilder>.

when we compare two boolean queries, we check that we have the same list of should clauses:

elasticsearch/server/src/main/java/org/elasticsearch/index/query/BoolQueryBuilder.java

Lines 333 to 340 in 5a0636a

	protected boolean doEquals(BoolQueryBuilder other) {
	return Objects.equals(adjustPureNegative, other.adjustPureNegative)
	&& Objects.equals(minimumShouldMatch, other.minimumShouldMatch)
	&& Objects.equals(mustClauses, other.mustClauses)
	&& Objects.equals(shouldClauses, other.shouldClauses)
	&& Objects.equals(mustNotClauses, other.mustNotClauses)
	&& Objects.equals(filterClauses, other.filterClauses);
	}

so here I couldn't just use a simple assertEquals check as we had before

[Mikep86]

Looks good, awesome work 🚀 ! I left some minor comments about tests, they should be very easy to address.

[Mikep86]

Can we check that rankWindowSize is propagated to the rewritten retriever as expected?

[Mikep86]

Nit: Use indexName instead of a magic string here? Repeat where "test-index" is referenced below.

[Mikep86]

Nit: Use anotherIndexName instead of a magic string here? Repeat where "test-another-index" is referenced below.