ES|QL - Add vector similarity functions to tech preview, allow pushdown of blockloader functions by carlosdelest · Pull Request #139365 · elastic/elasticsearch

carlosdelest · 2025-12-11T11:31:13Z

Get the following vector similarity functions into release builds:

V_DOT_PRODUCT
V_COSINE
V_L1_NORM
V_L2_NORM
V_HAMMING

Vector similarity functions use function pushdown to blockloaders - see #137002

Other functions that use pushdown to blockloaders are being pushed down as well.

…S_PUSHDOWN capability and use it on specific functions

elasticsearchmachine · 2025-12-11T11:32:21Z

Hi @carlosdelest, I've created a changelog YAML for you.

carlosdelest · 2025-12-11T11:37:17Z

...rc/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/PushExpressionToLoadIT.java

        );
    }

-    public void testVCosine() throws IOException {


Vector similarity functions tests have been moved to PushVectorSimilarityToLoadIT, as they don't need the check now. We should merge them together again when pushing down is not behind snapshot.

carlosdelest · 2025-12-11T11:38:22Z

...g/elasticsearch/xpack/esql/expression/function/scalar/convert/FromAggregateMetricDouble.java


    @Override
    public PushedBlockLoaderExpression tryPushToFieldLoading(SearchStats stats) {
+        if (EsqlCapabilities.Cap.BLOCK_LOADER_EXPRESSIONS_PUSHDOWN.isEnabled() == false) {


Checks pushdown capability on the pushdown for all the non-vector similarity functions

This is how I'd do it too, but I think we should enable all of these now.

carlosdelest · 2025-12-11T11:39:53Z

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

-                def(Magnitude.class, Magnitude::new, "v_magnitude"),
-                def(Hamming.class, Hamming::new, "v_hamming") } };
+                // dense vector functions
+                def(Magnitude.class, Magnitude::new, "v_magnitude")} };


V_MAGNITUDE is still not pushed down - #137535 tracks that

carlosdelest · 2025-12-11T11:41:34Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

+            new InferIsNotNull(),
+            new InferNonNullAggConstraint(),
+            new ReplaceDateTruncBucketWithRoundTo(),
+            new PushExpressionsToFieldLoad()


Add the PushExpressionsToFieldLoad rule by default - now the BlockLoaderExpression.tryPushToFieldLoading method needs to take into account pushdown capabilities

github-actions · 2025-12-11T11:47:17Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/functions-operators/dense-vector-functions.md

github-actions · 2025-12-11T11:47:18Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

elasticsearchmachine · 2025-12-11T14:38:35Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

ioanatia · 2025-12-11T15:51:09Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

+        checkVectorFunctionsNullArgs("v_magnitude(null)");
        if (EsqlCapabilities.Cap.MAGNITUDE_SCALAR_VECTOR_FUNCTION.isEnabled()) {
-            checkVectorFunctionsNullArgs("v_magnitude(null)");
-        }
-        if (EsqlCapabilities.Cap.HAMMING_VECTOR_SIMILARITY_FUNCTION.isEnabled()) {


I think something is not right here - shouldn't v_magnitude tests still be skipped for non snapshot builds and the v_hamming ones be executed without a cap check?

Oh I screwed up here, thanks for the catch!

I'm fixing in 21f8197, and adding the test-release label

…-tech-preview

carlosdelest · 2025-12-12T08:30:50Z

Failures are due to packaging and to #139401

…or-similarity-functions-tech-preview

…milarity-functions-tech-preview' into enhancement/esql-vector-similarity-functions-tech-preview

nik9000

Taking the vector similarity functions to preview looks right to me.

Enabling pushdown looks right. The per-function-opt-out looks right, though I'd prefer if we didn't do it. I'd save that for a follow-up PR, but I have naming objections to the new capability you used. And, I figure, fixing the name is only marginally less expensive than dropping it entirely.

I have no strong opinion on if you should keep PushVectorSimilarityToLoadIT or move everything back into the original file. You don't have to keep it if you remove the capability and globally enable the pushdowns. But you might like it for organizational purposes. After the split, PushExpressionToLoadIT does a lot of work around different shapes of queries and PushVectorSimilarityToLoadIT is much more readable.

nik9000 · 2025-12-12T13:49:01Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

+        /**
+         * Support for {@link org.elasticsearch.xpack.esql.expression.function.blockloader.BlockLoaderExpression} pushdown
+         */
+        BLOCK_LOADER_EXPRESSIONS_PUSHDOWN(Build.current().isSnapshot()),


Could you remove this entirely and enable all of the pushdowns? I think we're ready now.

nik9000 · 2025-12-12T13:49:36Z

...g/elasticsearch/xpack/esql/expression/function/scalar/convert/FromAggregateMetricDouble.java


    @Override
    public PushedBlockLoaderExpression tryPushToFieldLoading(SearchStats stats) {
+        if (EsqlCapabilities.Cap.BLOCK_LOADER_EXPRESSIONS_PUSHDOWN.isEnabled() == false) {


This is how I'd do it too, but I think we should enable all of these now.

carlosdelest · 2025-12-12T15:58:02Z

@nik9000 , I did allow pushdown for all functions and moved back PushVectorSimilarityToLoadIT to the original file in 067f542

…-tech-preview

…wn of blockloader functions (elastic#139365)

carlosdelest added 4 commits December 11, 2025 12:27

Remove capabilities where applicable, add new BLOCK_LOADER_EXPRESSION…

75cc1cf

…S_PUSHDOWN capability and use it on specific functions

Remove vector functions from snapshot

9f56d59

Add docs

3f40248

Separate vector similarity functions tests

f6b7338

elasticsearchmachine added the v9.3.0 label Dec 11, 2025

carlosdelest added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch :Search Relevance/ES|QL Search functionality in ES|QL >enhancement labels Dec 11, 2025

Update docs/changelog/139365.yaml

2f7dc4e

This was referenced Dec 11, 2025

ES|QL - Move dense vector similarity functions into release builds #138274

Closed

ES|QL - dense_vector exact (brute force) search support #130828

Closed

[CI] Auto commit changes from spotless

a1fa7d9

carlosdelest commented Dec 11, 2025

View reviewed changes

carlosdelest added 2 commits December 11, 2025 12:43

Fix changelog

f7fe954

Fix docs

a52412d

Fix hamming test

f7747ef

carlosdelest marked this pull request as ready for review December 11, 2025 14:38

elasticsearchmachine removed the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Dec 11, 2025

carlosdelest requested a review from nik9000 December 11, 2025 14:38

carlosdelest requested review from afoucret and ioanatia December 11, 2025 14:38

ioanatia reviewed Dec 11, 2025

View reviewed changes

Fix VerifierTests

21f8197

carlosdelest added the test-release Trigger CI checks against release build label Dec 12, 2025

Merge branch 'main' into enhancement/esql-vector-similarity-functions…

9c748a0

…-tech-preview

ioanatia approved these changes Dec 12, 2025

View reviewed changes

carlosdelest added 5 commits December 12, 2025 11:31

Merge remote-tracking branch 'origin/main' into enhancement/esql-vect…

2518ab2

…or-similarity-functions-tech-preview

Add FORK and subqueries CSV tests

f5721a9

Fix tests after merging with main

29285f1

Merge remote-tracking branch 'carlosdelest/enhancement/esql-vector-si…

1bd0851

…milarity-functions-tech-preview' into enhancement/esql-vector-similarity-functions-tech-preview

Fix capability

61d1a3f

nik9000 requested changes Dec 12, 2025

View reviewed changes

Allow all functions to be pushed down

067f542

carlosdelest requested a review from nik9000 December 12, 2025 15:57

[CI] Auto commit changes from spotless

bc4ad82

carlosdelest changed the title ~~ES|QL - Add vector similarity functions to tech preview~~ Dec 12, 2025

nik9000 added 2 commits December 12, 2025 15:13

Fix release tests

a7eae21

Merge branch 'main' into enhancement/esql-vector-similarity-functions…

9edf488

…-tech-preview

nik9000 approved these changes Dec 12, 2025

View reviewed changes

carlosdelest merged commit 99ced4f into elastic:main Dec 15, 2025
36 checks passed

ioanatia mentioned this pull request Dec 15, 2025

[ES|QL]: Update Vector Similarity To Support BFLOAT16 #139113

Merged

parkertimmins pushed a commit to parkertimmins/elasticsearch that referenced this pull request Dec 17, 2025

ES|QL - Add vector similarity functions to tech preview, allow pushdo…

3e0a4d2

…wn of blockloader functions (elastic#139365)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES|QL - Add vector similarity functions to tech preview, allow pushdown of blockloader functions#139365

ES|QL - Add vector similarity functions to tech preview, allow pushdown of blockloader functions#139365
carlosdelest merged 20 commits intoelastic:mainfrom
carlosdelest:enhancement/esql-vector-similarity-functions-tech-preview

carlosdelest commented Dec 11, 2025 •

edited

Loading

elasticsearchmachine commented Dec 11, 2025

carlosdelest Dec 11, 2025

carlosdelest Dec 11, 2025

nik9000 Dec 12, 2025

carlosdelest Dec 11, 2025

carlosdelest Dec 11, 2025

github-actions bot commented Dec 11, 2025 •

edited

Loading

github-actions bot commented Dec 11, 2025

When to use applies_to tags:

What NOT to do:

elasticsearchmachine commented Dec 11, 2025

ioanatia Dec 11, 2025

carlosdelest Dec 12, 2025

carlosdelest commented Dec 12, 2025

nik9000 left a comment •

edited

Loading

nik9000 Dec 12, 2025

nik9000 Dec 12, 2025

carlosdelest commented Dec 12, 2025

Uh oh!

Labels

4 participants

Conversation

carlosdelest commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticsearchmachine commented Dec 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

github-actions bot commented Dec 11, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

elasticsearchmachine commented Dec 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosdelest commented Dec 12, 2025

nik9000 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosdelest commented Dec 12, 2025

Uh oh!

Labels

4 participants

carlosdelest commented Dec 11, 2025 •

edited

Loading

github-actions bot commented Dec 11, 2025 •

edited

Loading

nik9000 left a comment •

edited

Loading