ES|QL: Pass fix size instead of maxPageSize to LuceneTopNOperator scorer

[ioanatia]

Initially we passed scorer.leafReaderContext().reader().maxDoc() to the bulk scorer, which showed great improvements.
However this will also keep the current thread busy until it is able to score all the docs from the current reader.
There are ways we can improve this, but they would require more changes.

For now I wanted to push a quick fix to mitigate some of the regressions we see, with the maxPageSize becoming smaller after #108412 .
We should still see a significant performance boost just by scoring docs in batches of 4096 docs, instead of maxPageSize.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[nik9000]

Do we need to lock this to the max of the range?

It looks like CancellableBulkScorer makes this bigger and bigger with time. But I think this is good and we can get it in and iterate.

[ioanatia]

We do something similar here, at least we try to avoid overflows:

elasticsearch/x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/lucene/LuceneOperator.java

Lines 241 to 246 in 7cbc236

	void scoreNextRange(LeafCollector collector, Bits acceptDocs, int numDocs) throws IOException {
	assert isDone() == false : "scorer is exhausted";
	// avoid overflow and limit the range
	numDocs = Math.min(maxPosition - position, numDocs);
	assert numDocs > 0 : "scorer was exhausted";
	position = bulkScorer.score(collector, acceptDocs, position, Math.min(maxPosition, position + numDocs));

[nik9000]

Ah. got it.

[kderusso]

LGTM, but consider labeling with >bug and creating a release note entry as this would be impacting serverless performance?

[carlosdelest]

Awesome finding! 🚤

[elasticsearchmachine]

Hi @ioanatia, I've created a changelog YAML for you.

[nik9000]

Ah. got it.

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	9.2