ESQL - Add byte element support for dense_vector data type by carlosdelest · Pull Request #131863 · elastic/elasticsearch

carlosdelest · 2025-07-24T17:27:57Z

Adds code and tests for supporting byte dense_vectors.

Internally, byte vectors get widened to floats. This is inefficient, but avoids having to create specific infrastructure for variable type blocks. This can be improved in the future.

…vector-byte-element-support # Conflicts: # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

carlosdelest · 2025-07-24T17:29:07Z

server/src/main/java/org/elasticsearch/index/mapper/BlockDocValuesReader.java

-        private final FloatVectorValues floatVectorValues;
-        private final KnnVectorValues.DocIndexIterator iterator;
-        private final int dimensions;
+    private abstract static class DenseVectorValuesBlockReader<T extends KnnVectorValues> extends BlockDocValuesReader {


Added abstract classes to deal with common code between float and byte vector reading

carlosdelest · 2025-07-24T17:30:10Z

server/src/main/java/org/elasticsearch/index/mapper/vectors/VectorEncoderDecoder.java

+     * @param vectorBR - dense vector encoded in BytesRef
+     * @param vector - array of bytes where the decoded vector should be stored
+     */
+    public static void decodeDenseVector(IndexVersion indexVersion, BytesRef vectorBR, byte[] vector) {


Needed a specific method for decoding dense vector of byte values - this is an adaptation of the existing float[] method (expand up to see it)

carlosdelest · 2025-07-24T17:30:34Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/dense_vector-byte.csv-spec

A copy of the float vector tests, using the specific byte field

carlosdelest · 2025-07-24T17:30:49Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/dense_vector.csv-spec


 FROM dense_vector
-| KEEP id, vector
+| KEEP id, float_vector


Renamed fields to avoid confusion to float_vector, byte_vector

carlosdelest · 2025-07-24T17:31:41Z

...n/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

        // Indexed field types
-        for (String indexType : DENSE_VECTOR_INDEX_TYPES) {
-            params.add(new Object[] { indexType, true, false });
+        for (String indexType : ALL_DENSE_VECTOR_INDEX_TYPES) {


Check all types for float and bytes. Quantized types only make sense for floats.

elasticsearchmachine · 2025-07-24T19:10:14Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

ioanatia

the approach makes sense to me, but there are some failures in KnnFunctionIT that we should look into

ioanatia · 2025-07-29T12:12:12Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

+        /**
+         * Byte elements dense vector field type support.
+         */
+        DENSE_VECTOR_FIELD_TYPE_BYTE_ELEMENTS(EsqlCorePlugin.DENSE_VECTOR_FEATURE_FLAG);


this make sense to me for now - when we get to take these features out of snapshot, maybe we can create single capability to encompass all the features, from dense_vector support to the knn function and brute force functions and we can remove the individual ones we have created for each step.

x-pack/plugin/esql/qa/testFixtures/src/main/resources/data/dense_vector.csv

tteofili

LGTM, great job Carlos!

…vector-byte-element-support

…r-support-normalization' into non-issue/esql-dense-vector-support-normalization # Conflicts: # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

… non-issue/esql-dense-vector-byte-element-support # Conflicts: # server/src/main/java/org/elasticsearch/index/mapper/BlockDocValuesReader.java # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/KnnFunctionIT.java

…r-byte-element-support' into non-issue/esql-dense-vector-byte-element-support # Conflicts: # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

…port

…r-byte-element-support' into non-issue/esql-dense-vector-byte-element-support

elasticsearchmachine · 2025-08-13T11:22:02Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

ChrisHegarty

This seems like a good first step. LGTM

…port

carlosdelest · 2025-08-14T10:47:06Z

Merging as CI failures are unrelated

…31863)

* upstream/main: (278 commits) ESQL - dense vector support cosine normalization (elastic#132721) [ML] Add support for dimensions in google vertex ai request (elastic#132689) ESQL - Add byte element support for dense_vector data type (elastic#131863) ESQL: Fix async operator warnings not always sent when blocking (elastic#132744) Method not needed anymore (elastic#132912) [Test] Excercise shutdown more reliably in snapshot stress IT (elastic#132909) Update Gradle shadow plugin to 9.0.1 (elastic#132637) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search/410_named_queries/named_queries_with_score} elastic#132906 Update docker.elastic.co/wolfi/chainguard-base-fips:latest Docker digest to fa6cb69 (elastic#132735) Remove unnecessary calls to fold() (elastic#131870) Use consistent terminology for transport version resources/references (elastic#132882) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search.vectors/40_knn_search_cosine/kNN search only regular query} elastic#132890 Finalize release notes for v9.1.2 release (elastic#132745) Finalize release notes for v9.0.5 release (elastic#132718) Move inner records out of TransportVersionUtils (elastic#132872) Add support for Lookup Join on Multiple Fields (elastic#131559) Bootstrap PR-based benchmarks (elastic#132717) Refactor MetadataIndexTemplateService to use template maps instead of project metadata (elastic#132662) [Gradle] Update nebula ospackage plugin to 12.1.0 (elastic#132640) Mute org.elasticsearch.xpack.esql.CsvTests test {csv-spec:ip.CdirMatchEqualsInsOrs} elastic#132860 ...

carlosdelest added 7 commits July 24, 2025 17:01

Add byte vector support

a62c943

Add byte vector support - tests

9317fcd

First test version, with all index types - fails

4c59ba1

Knn tests for non-flat, indexed types

475a285

Add CSV tests

ecc563f

Merge remote-tracking branch 'origin/main' into non-issue/esql-dense-…

4a40503

…vector-byte-element-support # Conflicts: # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

Fix tests after merging

84d3c50

carlosdelest added >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch v9.2.0 labels Jul 24, 2025

carlosdelest commented Jul 24, 2025

View reviewed changes

[CI] Auto commit changes from spotless

e9afc06

carlosdelest marked this pull request as ready for review July 24, 2025 19:09

elasticsearchmachine removed the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Jul 24, 2025

carlosdelest requested review from ChrisHegarty, afoucret, ioanatia, svilen-mihaylov-elastic and tteofili July 24, 2025 19:10

ioanatia reviewed Jul 29, 2025

View reviewed changes

tteofili mentioned this pull request Jul 29, 2025

V_HAMMING – calculates Hamming distance #132056

Closed

tteofili approved these changes Jul 29, 2025

View reviewed changes

carlosdelest marked this pull request as draft August 6, 2025 15:14

carlosdelest and others added 4 commits August 11, 2025 17:13

Merge remote-tracking branch 'origin/main' into non-issue/esql-dense-…

0402ad9

…vector-byte-element-support

Take into account normalization

a4aca14

[CI] Auto commit changes from spotless

08a3c5c

Take into account normalization for dense vector support

763fe63

carlosdelest and others added 16 commits August 12, 2025 14:20

Fix cherry pick

80b48cf

[CI] Auto commit changes from spotless

40edca3

Remove debugging code

8bd7f79

Merge remote-tracking branch 'carlosdelest/non-issue/esql-dense-vecto…

f9447f7

…r-support-normalization' into non-issue/esql-dense-vector-support-normalization # Conflicts: # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

Check that we may not have magnitudes at all, or for normalized vectors

7d2625c

Fix merge

5bcac49

Remove cosine similarity code

57c45b5

Merge remote-tracking branch 'carlosdelest/non-issue/esql-dense-vecto…

9abc1ea

…r-byte-element-support' into non-issue/esql-dense-vector-byte-element-support # Conflicts: # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/DenseVectorFieldTypeIT.java

[CI] Auto commit changes from spotless

6977dbe

Merge branch 'main' into non-issue/esql-dense-vector-byte-element-sup…

7675c71

…port

Better parameterized test

239d350

Fix test

9385113

Merge remote-tracking branch 'carlosdelest/non-issue/esql-dense-vecto…

67cbb84

…r-byte-element-support' into non-issue/esql-dense-vector-byte-element-support

Fix test

14ba3b0

[CI] Auto commit changes from spotless

64ca563

carlosdelest marked this pull request as ready for review August 13, 2025 10:51

carlosdelest added Team:ES|QL :Search Relevance/ES|QL Search functionality in ES|QL labels Aug 13, 2025

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Aug 13, 2025

ChrisHegarty approved these changes Aug 13, 2025

View reviewed changes

Merge branch 'main' into non-issue/esql-dense-vector-byte-element-sup…

ee803a4

…port

carlosdelest merged commit 26ffd7f into elastic:main Aug 14, 2025
31 of 34 checks passed

carlosdelest mentioned this pull request Aug 14, 2025

ES|QL dense vector support (GA) #125783

Closed

7 tasks

joshua-adams-1 pushed a commit to joshua-adams-1/elasticsearch that referenced this pull request Aug 14, 2025

ESQL - Add byte element support for dense_vector data type (elastic#1…

eefd3a3

…31863)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESQL - Add byte element support for dense_vector data type#131863

ESQL - Add byte element support for dense_vector data type#131863
carlosdelest merged 29 commits intoelastic:mainfrom
carlosdelest:non-issue/esql-dense-vector-byte-element-support

carlosdelest commented Jul 24, 2025

carlosdelest Jul 24, 2025

carlosdelest Jul 24, 2025

carlosdelest Jul 24, 2025

carlosdelest Jul 24, 2025

carlosdelest Jul 24, 2025

elasticsearchmachine commented Jul 24, 2025

ioanatia left a comment

ioanatia Jul 29, 2025

Uh oh!

tteofili left a comment

elasticsearchmachine commented Aug 13, 2025

ChrisHegarty left a comment

carlosdelest commented Aug 14, 2025

Uh oh!

Labels

5 participants

Conversation

carlosdelest commented Jul 24, 2025

carlosdelest Jul 24, 2025

Choose a reason for hiding this comment

carlosdelest Jul 24, 2025

Choose a reason for hiding this comment

carlosdelest Jul 24, 2025

Choose a reason for hiding this comment

carlosdelest Jul 24, 2025

Choose a reason for hiding this comment

carlosdelest Jul 24, 2025

Choose a reason for hiding this comment

elasticsearchmachine commented Jul 24, 2025

ioanatia left a comment

Choose a reason for hiding this comment

ioanatia Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

tteofili left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented Aug 13, 2025

ChrisHegarty left a comment

Choose a reason for hiding this comment

carlosdelest commented Aug 14, 2025

Uh oh!

Labels

5 participants