Auto prefiltering for queries on dense semantic_text fields by dimitris-athanasiou · Pull Request #138989 · elastic/elasticsearch

dimitris-athanasiou · 2025-12-03T16:30:56Z

knn queries allow specifying filters that will be applied before the knn search. This pre-filtering allows the knn to return k results. If such filters are to applied only after the knn executes, then the knn returns the k matching results but the filters can filter out some of them thus potentially returning fewer than k results.

semantic_text fields can be queried with:

DSL

match queries
semantic queries
knn queries

ES|QL

match queries
knn queries

For DSL, knn queries allow users to specify direct prefilters. However, match and semantic queries provide no way to do so. Same goes for ES|QL MATCH. Noting that ES|QL KNN already implements auto pre-filtering where conjunctions are pushed down to the knn query as prefilters.

This commit implements semantic_text auto pre-filtering for semantic_text queries in DSL (match and semantic queries) and ES|QL (MATCH).

We achieve this by adding an AutoPrefilteringScope object to the SearchExecutionContext. When we convert a bool query to a lucene query, we push its must, filter, and must_not clauses to the AutoPrefilteringScope. At that stage queries have already been rewritten. Semantic queries using text_embedding inference endpoints are rewritten to knn vector queries that are auto-prefiltering enabled. Then, when an auto-prefiltering enabled knn vector query is converted to its lucene equivalent, we fetch the prefilters from the SearchExecutionContext and we apply them to the knn vector query - which supports pre-filtering already.

ES|QL queries that contain MATCH automatically benefit from this implementation because they are rewritten in bool queries.

Limitations

DSL

nested queries are excluded from pre-filtering (nested KNN query returns no results when using nested query on top level field as filter #138184)

ES|QL

filters that are not translatable to lucene queries will be applied as post-filters

Relates #132068

`knn` queries allow specifying `filters` that will be applied before the knn search. This `pre-filtering` allows the `knn` to return `k` results. If such filters are to applied only after the `knn` executes, then the `knn` returns the `k` matching results but the filters can filter out some of them thus potentially returning fewer than `k` results. `semantic_text` fields can be queried with: DSL - `match` queries - `semantic` queries - `knn` queries ES|QL - `match` queries - `knn` queries For DSL, `knn` queries allow users to specify direct prefilters. However, `match` and `semantic` queries provide no way to do so. Same goes for ES|QL `match`. Noting that ES|QL `KNN` already implements auto pre-filtering where conjunctions are pushed down to the `knn` query as prefilters. This commit implements semantic_text auto pre-filtering for `semantic_text` queries in DSL (`match` and `semantic` queries) and ES|QL (`MATCH`). We achieve this by adding an `AutoPrefilteringScope` object to the `SearchExecutionContext`. When we convert a `bool` query to a lucene query, we push its `must`, `filter`, and `must_not` clauses to the `AutoPrefilteringScope`. At that stage queries have already been rewritten. Semantic queries using `text_embedding` inference endpoints are rewritten to knn vector queries that are auto-prefiltering enabled. Then, when an auto-prefiltering enabled knn vector query is converted to its lucene equivalent, we fetch the prefilters from the `SearchExecutionContext` and we apply them to the knn vector query - which supports pre-filtering already. ES|QL queries that contain `MATCH` automatically benefit from this implementation because they are rewritten in `bool` queries. Limitations DSL - nested queries are excluded from pre-filtering (elastic#138184) ES|QL - filters that are not translatable to lucene queries will be applied as post-filters Relates elastic#132068

elasticsearchmachine · 2025-12-03T16:31:22Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

elasticsearchmachine · 2025-12-03T16:31:22Z

Hi @dimitris-athanasiou, I've created a changelog YAML for you.

server/src/main/java/org/elasticsearch/index/query/support/AutoPrefilteringScope.java

...esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/SemanticMatchTestCase.java

Mikep86

Partial review, nice work! I didn't get to the tests, but I reviewed all the production logic. I identified some potential edge cases that I think we could iterate on.

server/src/main/java/org/elasticsearch/index/query/BoolQueryBuilder.java

server/src/main/java/org/elasticsearch/search/vectors/KnnVectorQueryBuilder.java

Mikep86 · 2025-12-08T22:04:52Z

server/src/main/java/org/elasticsearch/search/vectors/KnnVectorQueryBuilder.java

                rescoreVectorBuilder,
                vectorSimilarity
-            ).boost(boost).queryName(queryName).addFilterQueries(filterQueries);
+            ).boost(boost).queryName(queryName).addFilterQueries(filterQueries).setAutoPrefilteringEnabled(isAutoPrefilteringEnabled);


There's a bunch of places in the query interception logic where we need to make a copy of the knn query, except slightly tweaked. It's very easy to overlook the need to call setAutoPrefilteringEnabled when making such copies. Maybe it's time for a little static helper method that takes an origin knn query and applies boost, queryName, and autoPrefilteringEnabled values to a target knn query?

To be fair the constructor situation in KnnVectorQueryBuilder has gone wild and I agree it is very fragile. A static helper would be nice but we'd still need to remember to call it. I wonder if the right solution here is a refactoring of the constructors. How about we leave this is follow up work? I can raise an issue for tidying this up.

Agreed a more thorough refactoring is needed here, which we can do in a follow-up. IMO we should refactor KnnVectorQueryBuilder to use a builder pattern that can take an existing KnnVectorQueryBuilder to initialize.

dimitris-athanasiou · 2025-12-10T12:01:32Z

@Mikep86 I have pushed commits to the PR where I address your feedback:

I have added a AutoPrefilteringUtils.pruneQuery that takes in a set of QueryBuilder classes, looks into the query tree and prunes query branches if from the query that matches one of the given classes. We only use NestedQueryBuilder for now.
I have removed the loop protection in KnnVectorQueryBuilder as it is no longer necessary as we ensure queries exclude themselves from becoming prefilters in their inner queries.

Mikep86

Fantastic work 🙌 ! All the production code looks good. I pointed out a potential edge case in the min-should-match handing, but I don't have a good solution for it (it's also a very narrow edge case). Other than that, it's just a few small adjustments to tests.

server/src/main/java/org/elasticsearch/index/query/support/AutoPrefilteringUtils.java

server/src/test/java/org/elasticsearch/index/query/BoolQueryBuilderTests.java

...er/src/test/java/org/elasticsearch/search/vectors/AbstractKnnVectorQueryBuilderTestCase.java

.../yamlRestTest/resources/rest-api-spec/test/inference/100_semantic_text_auto_prefiltering.yml

Mikep86

LGTM 🚀

Mikep86 · 2025-12-11T19:26:31Z

server/src/main/java/org/elasticsearch/index/query/support/AutoPrefilteringUtils.java

+            // We need to adjust the minimum should match to account for the pruned clauses.
+            // We considered the following approaches:
+            // 1. strict approach: set to min(remaining_should_clauses, original_msm)
+            // 2. lenient approach: if msm is set and at least one should clause is pruned, prune all should clauses.
+            // 3. middle ground approach: set to max(0, original_msm - remaining_should_clauses)
+            // Let us imagine a query with 5 should clauses. 2 get pruned. msm is 3. 1 remaining clause matches.
+            // Approach 1 would make the entire bool query to not match as we would retain msm of 3 but only 1 clause would match.
+            // We do not know whether the pruned clauses would match or not. Thus, this approach seems too restrictive.
+            // Approach 2 would mean we prune all should clauses and the query would match,
+            // even if none of the remaining should clauses match.
+            // Approach 3 would mean we adjust the msm to 3 - 2 = 1. This would mean that the query would match if at least one
+            // of the remaining clauses matches.
+            // We opt for the lenient approach. It is as if we assume the pruned clauses matched. Seems to be the best compromise.


Thank you for the thorough description ❤️

carlosdelest

LGTM, amazing work!

It would be awesome to have a test in SemanticMatchTestCase that checks that ES|QL applies prefiltering - but not needed for this PR

carlosdelest · 2025-12-12T07:55:10Z

server/src/main/java/org/elasticsearch/index/query/support/AutoPrefilteringUtils.java

+            return Optional.empty();
+        }
+
+        if (query instanceof BoolQueryBuilder boolQuery) {


Nit - Consider using pattern matching for switch

Done in 91c0a75. Much prettier!

carlosdelest · 2025-12-12T08:21:09Z

...er/src/test/java/org/elasticsearch/search/vectors/AbstractKnnVectorQueryBuilderTestCase.java

    }

+    public void testBWCVersionSerialization_GivenAutoPrefiltering() throws IOException {
+        for (int i = 0; i < 100; i++) {


Why execute this multiple times? Is this a loop for testing the test?

During my testing I found that I needed that to surface problems faster. It runs pretty fast so I left it in.

You should use -Dtests.iters= instead 😉 . Let's remove this as you'll get plenty of executions on CI anyway.

That is true. But if you take a look at AbstractBWCSerializationTestCase you'll see we also do multiple runs there by default. What it helps with is that if someone makes a change that breaks BWC, they might run the tests once, they pass and they think it's all good. Whereas running a bunch of times significantly increases the probability to surface a failure and gives immediate feedback to the dev to fix the issue before getting in CI.

I was not aware of that, thanks!

Maybe then use NUMBER_OF_TEST_RUNS instead to keep with the pattern? 🤷

Done in 1a8cdc2

AbstractQueryTestCase has its own such constant, NUMBER_OF_TESTQUERIES

dimitris-athanasiou · 2025-12-12T08:42:48Z

It would be awesome to have a test in SemanticMatchTestCase that checks that ES|QL applies prefiltering - but not needed for this PR

@carlosdelest I have added such a test! It's there!

carlosdelest · 2025-12-12T08:46:23Z

It would be awesome to have a test in SemanticMatchTestCase that checks that ES|QL applies prefiltering - but not needed for this PR

@carlosdelest I have added such a test! It's there!

@dimitris-athanasiou It indeed is! Isn't that awesome? 😅 🤦

Adds documentation for automatic pre-filtering that was introduced in elastic#138989.

Adds documentation for automatic pre-filtering that was introduced in #138989. Co-authored-by: Liam Thompson <leemthompo@gmail.com>

dimitris-athanasiou added >bug :SearchOrg/Relevance Label for the Search (solution/org) Relevance team v9.3.0 labels Dec 3, 2025

elasticsearchmachine added the Team:Search - Relevance The Search organization Search Relevance team label Dec 3, 2025

Update docs/changelog/138989.yaml

55f0447

This was referenced Dec 3, 2025

POC - Automatic prefiltering for semantic_text queries #137467

Closed

POC-2 - Auto prefiltering for semantic text queries #137739

Closed

dimitris-athanasiou added 6 commits December 4, 2025 13:25

Changelog -> Vector Search

d5976c8

Sort by score in ES|QL test

37415ac

Merge branch 'main' into semantic_text_auto_prefiltering

3ff031d

Make keyword field a filter clause in ES|QL test to exclude from scoring

189c739

Merge branch 'main' into semantic_text_auto_prefiltering

7c27a19

Merge branch 'main' into semantic_text_auto_prefiltering

27f736d

ioanatia reviewed Dec 8, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/index/query/support/AutoPrefilteringScope.java Show resolved Hide resolved

ioanatia reviewed Dec 8, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/index/query/support/AutoPrefilteringScope.java Show resolved Hide resolved

...esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/SemanticMatchTestCase.java Show resolved Hide resolved

dimitris-athanasiou added 2 commits December 8, 2025 17:32

Add better javadoc to AutoPrefilteringScope

e803311

Add a YAML test for semantic query

f5563b3

Mikep86 requested a review from a team December 8, 2025 20:58

Mikep86 reviewed Dec 8, 2025

View reviewed changes

dimitris-athanasiou added 4 commits December 9, 2025 18:16

Prune auto-prefilters

371bd0a

No need for loop protection while applying auto-prefiltering

2755047

Tests for pruning

16d40e4

Merge branch 'main' into semantic_text_auto_prefiltering

87431e1

dimitris-athanasiou added 3 commits December 10, 2025 14:31

Missing null check for minimum_should_match

fbe40a2

Merge branch 'main' into semantic_text_auto_prefiltering

60c4e59

Merge branch 'main' into semantic_text_auto_prefiltering

400a069

Mikep86 reviewed Dec 10, 2025

View reviewed changes

dimitris-athanasiou added 6 commits December 11, 2025 13:18

Address review feedback on tests

a18751f

Merge branch 'main' into semantic_text_auto_prefiltering

b066bd4

Better handling of minimum_should_match

f4451e8

Merge branch 'main' into semantic_text_auto_prefiltering

f1c2793

Merge branch 'main' into semantic_text_auto_prefiltering

fc8562c

Merge branch 'main' into semantic_text_auto_prefiltering

15eec53

Mikep86 approved these changes Dec 11, 2025

View reviewed changes

Merge branch 'main' into semantic_text_auto_prefiltering

f622550

carlosdelest approved these changes Dec 12, 2025

View reviewed changes

ioanatia approved these changes Dec 12, 2025

View reviewed changes

dimitris-athanasiou added 4 commits December 12, 2025 10:49

Use pattern matching for switch

91c0a75

Use constant for number of test runs for BWC serialization

1a8cdc2

Merge branch 'main' into semantic_text_auto_prefiltering

dd2945a

Merge branch 'main' into semantic_text_auto_prefiltering

844dd7a

dimitris-athanasiou merged commit d8b6b9c into elastic:main Dec 12, 2025
35 checks passed

This was referenced Dec 12, 2025

ES|QL: automatic prefiltering for semantic_text match queries #132068

Closed

[ES|QL] Semantic (dense) MATCH queries with filters that aren't translatable to Lucene apply post knn query #139453

Open

dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this pull request Dec 18, 2025

Adds documentation for semantic_text auto pre-filtering

5b28949

Adds documentation for automatic pre-filtering that was introduced in elastic#138989.

dimitris-athanasiou mentioned this pull request Dec 18, 2025

Documentation for semantic_text auto pre-filtering #139749

Merged

dimitris-athanasiou added a commit that referenced this pull request Dec 19, 2025

Documentation for semantic_text auto pre-filtering (#139749)

5d9ed4d

Adds documentation for automatic pre-filtering that was introduced in #138989. Co-authored-by: Liam Thompson <leemthompo@gmail.com>

Conversation

dimitris-athanasiou commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Limitations

elasticsearchmachine commented Dec 3, 2025

elasticsearchmachine commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimitris-athanasiou commented Dec 10, 2025

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosdelest left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimitris-athanasiou commented Dec 12, 2025 • edited by carlosdelest Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

carlosdelest commented Dec 12, 2025

Uh oh!

Labels

5 participants

dimitris-athanasiou commented Dec 3, 2025 •

edited

Loading

dimitris-athanasiou commented Dec 12, 2025 •

edited by carlosdelest

Loading