Improve SingleValueMatchQuery performance

[martijnvg]

Changes:

1. If a field is single-valued and dense, then rewrite to match all docs. This logic already exists for when a field has points or inverted index. The logic can now also work with doc values only using the docIDRunEnd() method. This method will return maxDoc if field is dense.
2. Allow SingleValueMatchQuery to be cached if a field is single-valued, given that it will never emit a warning.

Note that the first change also allows WHERE clauses for single valued fields to be cached again.

Running the following without this change:

repeat 50000 curl -s -k -XPOST -H "Content-Type: application/json" -d '{"query": "FROM hits | WHERE AdvEngineID == 0 | STATS count = COUNT(*)"}' "http://localhost:9200/_query"  | jq .took

945
778
739
720
892
524
521
517
517
520
517
518
520
517
517
521
517
519

Running the following with the change:

repeat 50000 curl -s -k -XPOST -H "Content-Type: application/json" -d '{"query": "FROM hits | WHERE AdvEngineID == 0 | STATS count = COUNT(*)"}' "http://localhost:9200/_query"  | jq .took

432
249
247
247
377
5
5
5
5
4
5
4
4
4
4
4
4
4
4
4
4
4
4

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Hi @martijnvg, I've created a changelog YAML for you.

[nik9000]

Very nice! I don't know the lucene bits well enough these days to approve it, but I could reread if you need it. Otherwise, maybe @iverase can look too?

[iverase]

very nice

[dnhatn]

@martijnvg I have one question, but these changes will be helpful! Thanks Martijn.

[dnhatn]

I might be missing something, but I think checking if the singleton is not null is sufficient. We don't need to verify that all documents have values to return match_all.

[martijnvg]

This was my thinking as well, however then tests started to fail. I think this is because the query is supposed to returns only the docs with exactly one value. Also this is inline with the logic that checks points and terms (detecting that field is dense).

[dnhatn]

I think this is because the query is supposed to returns only the docs with exactly one value

Yes, if we have a singleton, we should be able to shortcut to match_all. I think there might be an issue with the test.

[martijnvg]

Nhat and I discussed in Slack:

the contract of the SingleValueMatchQuery query is that it only filters out docs that don't have exactly one value. If we were only to check whether doc values is a singleton during query rewrite, then that would break that contract for sparse dense fields (since also doc ids with zero values are included).

The plan is to do the rewrite to match only query if fields are singleton change in a followup change. Given that this changes to contract of SingleValueMatchQuery, but that shouldn't be an issue for es|ql. The contract would then be to exclude doc ids with more than 1 value.