Adding check for isIndexed in text fields when generating field exists queries to avoid ISE when field is stored but not indexed or with doc_values

[pmpailis]

As per #129973 (comment), it could be possible that if a text field is not indexed, but stored (i.e.

                    "b_field": {
                        "type": "text",
                        "index": false,
                        "store": true
                    }

we would be met with the following ISE exception from Lucene

Caused by: java.lang.IllegalStateException: FieldExistsQuery requires that the field indexes doc values, norms or vectors, but field 'b_field' exists and indexes neither of these data structures
at org.apache.lucene.core@10.2.1/org.apache.lucene.search.FieldExistsQuery.rewrite(FieldExistsQuery.java:156)
at org.apache.lucene.core@10.2.1/org.apache.lucene.search.ConstantScoreQuery.rewrite(ConstantScoreQuery.java:44)
at org.apache.lucene.core@10.2.1/org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:311)
at org.apache.lucene.core@10.2.1/org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:316)
at org.apache.lucene.core@10.2.1/org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:876)

As per @martijnvg description in the linked issue (thank you :) ):

the issue is that FieldExistsQuery gets used because TextSearchInfo indicates that norms are enabled. However because no inverted index is stored (index is false) we assume that there are norms and use FieldExistsQuery in MappedFieldType#existsQuery(). However FieldExistsQuery can only be returned if the field is also indexed and has norms.

Closes: #129973

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @pmpailis, I've created a changelog YAML for you.

[martijnvg]

LGTM 👍

[pmpailis]

There is a test failing in KeywordFieldTypeTests that I try to understand if it's testing a viable scenario, and whether the introduced change might actually affect results.

        {
            FieldType fieldType = new FieldType();
            fieldType.setOmitNorms(false);
            KeywordFieldType ft = new KeywordFieldType("field", fieldType);
            assertEquals(new FieldExistsQuery("field"), ft.existsQuery(MOCK_CONTEXT));
        }

The above now fails (we instead get a TermQuery) because we have the default doc_values: false, and index: false, whereas we explicitly state that norms: true. So, by adding the isIndexed check as a precondition for checking norms, we now force the mapper to generate a TermQuery instead. It's also worth noting that the ctor with the custom FieldType is only used from the test.

FWIW, adding the isIndexed precondition also prevents an ISE coming from a mapping as follows (that mimics the above with the addition of store: true):

            "keyword": {
                "type": "keyword",
                "index": false,
                "doc_values": false,
                "store": true,
                "norms": true
            }

I guess it's ok to update the test to the new generated query (and add a yaml test to ensure that we do get back results in other cases) ?

[martijnvg]

I guess it's ok to update the test to the new generated query (and add a yaml test to ensure that we do get back results in other cases) ?

I think so, given that this case was also prone to ISE.

[pmpailis]

Updated tests in 7d05df9df3bdc8e6192f979b0f394c8a6314c697