Skip to content

ES|QL - add dense_vector field type and to_dense_vector function to release builds#135604

Merged
carlosdelest merged 40 commits intoelastic:mainfrom
carlosdelest:feature/esql-dense-vector-tech-preview
Sep 30, 2025
Merged

ES|QL - add dense_vector field type and to_dense_vector function to release builds#135604
carlosdelest merged 40 commits intoelastic:mainfrom
carlosdelest:feature/esql-dense-vector-tech-preview

Conversation

@carlosdelest
Copy link
Member

@carlosdelest carlosdelest commented Sep 29, 2025 •
edited
Loading

Take dense_vector field type an TO_DENSE_VECTOR function out of snapshot builds.

This means that support must be explicitly removed for some functions and operators:

  • Aggregation functions
  • Multivalued functions - makes no sense as dense_vector is internally represented as a multivalued float

It is alsosupported by the following functions:

  • Is null / Is not null
  • CASE
  • TO_STRING
  • Equals / Not equals

Feature flag is removed, and capability checks removed where applicable.

Docs have been updated for the above functions, vector similarity functions, and knn.
@carlosdelest carlosdelest added ES|QL-ui Impacts ES|QL UI Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:Search - Relevance The Search organization Search Relevance team v9.2.0 Team:ES|QL :Search Relevance/ES|QL Search functionality in ES|QL labels Sep 29, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 29, 2025

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?
@carlosdelest carlosdelest marked this pull request as ready for review September 30, 2025 05:48
@elasticsearchmachine elasticsearchmachine removed the Team:Search - Relevance The Search organization Search Relevance team label Sep 30, 2025
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Sep 30, 2025

Pinging @elastic/es-search-relevance (Team:Search Relevance)
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Sep 30, 2025

Pinging @elastic/kibana-esql (ES|QL-ui)
@carlosdelest carlosdelest changed the title ES|QL - add dense_vector field type and to_dense_vector function Sep 30, 2025
alex-spies
alex-spies approved these changes Sep 30, 2025
View reviewed changes
Copy link
Contributor

@alex-spies alex-spies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - I have some minor, non-blocking comments.

I hate to say it, but we probably need to apply the test-release and test-full-bwc labels and see what additional test failures we may now get.
.../plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/plugin/EsqlCorePlugin.java
Comment on lines 15 to -17

public static final FeatureFlag AGGREGATE_METRIC_DOUBLE_FEATURE_FLAG = new FeatureFlag("esql_aggregate_metric_double");
public static final FeatureFlag DENSE_VECTOR_FEATURE_FLAG = new FeatureFlag("esql_dense_vector");
Copy link
Contributor

@alex-spies alex-spies Sep 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

I don't think we will use feature flags for under construction types going forward. We don't ask remote nodes if they have the same feature flag enabled, and don't have a good mechanism (yet) for when they don't.
.../src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/ToString.java
}

@ConvertEvaluator(extraName = "FromFloat")
static BytesRef fromFloat(float flt) {
Copy link
Contributor

@alex-spies alex-spies Sep 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this now?
Copy link
Member Author

@carlosdelest carlosdelest Sep 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dense vectors are internally represented as Float blocks - I don't think we had a valid ToString on floats as they are not representable AFAIU
Copy link
Contributor

@alex-spies alex-spies Sep 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, okay. The javadoc on the type says it's arrays of doubles, which got me confused. Thanks!
...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java
Comment on lines 543 to 547
def(Knn.class, tri(Knn::new), "knn"),
def(CosineSimilarity.class, CosineSimilarity::new, "v_cosine"),
def(DotProduct.class, DotProduct::new, "v_dot_product"),
def(L1Norm.class, L1Norm::new, "v_l1_norm"),
def(L2Norm.class, L2Norm::new, "v_l2_norm"),
Copy link
Contributor

@alex-spies alex-spies Sep 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess these will be released in a separate PR?
Copy link
Member Author

@carlosdelest carlosdelest Sep 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we want to improve performance before releasing vector similarity functions
...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/ExpressionWritables.java
entries.add(ToDatetime.ENTRY);
entries.add(ToDateNanos.ENTRY);
entries.add(ToDegrees.ENTRY);
if (EsqlCapabilities.Cap.TO_DENSE_VECTOR_FUNCTION.isEnabled()) {
Copy link
Contributor

@alex-spies alex-spies Sep 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice that you're cleaning this up, too. Not technically necessary, but nice.
@carlosdelest carlosdelest added test-full-bwc Trigger full BWC version matrix tests test-release Trigger CI checks against release build labels Sep 30, 2025
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Sep 30, 2025

Hi @carlosdelest, I've updated the changelog YAML for you.
carlosdelest and others added 2 commits September 30, 2025 12:46
@carlosdelest
Update docs/changelog/135604.yaml
2ed9e8b
Merge remote-tracking branch 'origin/main' into feature/esql-dense-ve…
46dabc2 
…ctor-tech-preview

# Conflicts:
#	server/src/main/resources/transport/upper_bounds/9.2.csv
@carlosdelest carlosdelest enabled auto-merge (squash) September 30, 2025 10:53
@alex-spies alex-spies mentioned this pull request Sep 30, 2025
Merged
elasticsearchmachine added 2 commits September 30, 2025 14:12
@carlosdelest carlosdelest removed test-full-bwc Trigger full BWC version matrix tests test-release Trigger CI checks against release build labels Sep 30, 2025
@carlosdelest carlosdelest merged commit 771aaff into elastic:main Sep 30, 2025
33 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ES|QL-ui Impacts ES|QL UI >feature :Search Relevance/ES|QL Search functionality in ES|QL Team:ES|QL Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch v9.2.0

5 participants

@carlosdelest @elasticsearchmachine @ioanatia @afoucret @alex-spies