Skip to content

ES|QL: Fix VectorSimilarityFunctionsIT#133206

Merged
ioanatia merged 1 commit intoelastic:mainfrom
ioanatia:fix_vector_similarity_test
Aug 20, 2025
Merged

ES|QL: Fix VectorSimilarityFunctionsIT#133206
ioanatia merged 1 commit intoelastic:mainfrom
ioanatia:fix_vector_similarity_test

Conversation

@ioanatia
Copy link
Contributor

@ioanatia ioanatia commented Aug 20, 2025

fixes #133059

Error details:

The tests fail with this type of error:

VectorSimilarityFunctionsIT > testSimilarityBetweenVectors {functionName=v_l2_norm similarityFunction=org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT$$Lambda/0x00000e00003db5d0@48df4071} FAILED
    java.lang.IllegalStateException: Block [FloatVectorBlock[vector=ConstantFloatVector[positions=0, value=0.0]]] does not have same position count: 0 != 1
        at org.elasticsearch.compute.data.Page.<init>(Page.java:88)
        at org.elasticsearch.compute.data.Page.appendBlock(Page.java:162)
        at org.elasticsearch.compute.operator.EvalOperator.process(EvalOperator.java:46)
        at org.elasticsearch.compute.operator.AbstractPageMappingOperator.getOutput(AbstractPageMappingOperator.java:85)
        at org.elasticsearch.compute.operator.Driver.runSingleLoopIteration(Driver.java:272)
        at org.elasticsearch.compute.operator.Driver.run(Driver.java:186)
        at org.elasticsearch.compute.operator.Driver$1.doRun(Driver.java:420)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
        at org.elasticsearch.compute.operator.DriverScheduler$1.doRun(DriverScheduler.java:57)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1067)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
        at java.base/java.lang.Thread.run(Thread.java:1447)

when this happens, we are not releasing the blocks so the next tests fail with :

    java.lang.AssertionError: Request breaker not reset to 0 on node: node_s0
    Expected: <0L>
         but: was <64L>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)

Fix

In the VectorSimilarityFunction evaluator, we first try and detect the vector dimensions by checking the left vector.
We need the vector dimension to instantiate the leftScratch and rightScratch float arrays which we reuse for each row.
In the case where the leftVector has only null values (0 dimensions) - we can return early, since in this case the similarity function should just return null values (as per #132919).
This is where the bug resides, we return a constant float block with a single position (and 0 value, not null).
Changing this to return a constant null block with the right positionCount should fix the issue.
@ioanatia ioanatia added >test Issues or PRs that are addressing/adding tests :Analytics/ES|QL AKA ESQL :SearchOrg/Relevance Label for the Search (solution/org) Relevance team Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:Search - Relevance The Search organization Search Relevance team v9.2.0 :Search Relevance/ES|QL Search functionality in ES|QL labels Aug 20, 2025
@elasticsearchmachine elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:SearchOrg Meta label for the Search Org (Enterprise Search) labels Aug 20, 2025
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Aug 20, 2025

Pinging @elastic/es-analytical-engine (Team:Analytics)
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Aug 20, 2025

Pinging @elastic/es-search-relevance (Team:Search Relevance)
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Aug 20, 2025

Pinging @elastic/search-eng (Team:SearchOrg)
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Aug 20, 2025

Pinging @elastic/search-relevance (Team:Search - Relevance)
@ioanatia ioanatia mentioned this pull request Aug 20, 2025
Closed
@ioanatia ioanatia merged commit 15e9d05 into elastic:main Aug 20, 2025
34 checks passed
szybia added a commit to szybia/elasticsearch that referenced this pull request Aug 20, 2025
@szybia
Merge remote-tracking branch 'upstream/main' into geoip-enrich-cache-…
b810b0d 
…improv

* upstream/main: (58 commits)
  Fixing flaky LoggedExec (tests) (elastic#133215)
  CPS search should not use `skip_unavailable` (elastic#132927)
  Don't fail search if bottom doc can't be formatted (elastic#133188)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupByNothing elastic#133225
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedSetDocValuesWithSkipperSmall elastic#133224
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedNumberMergeAwayAllValuesWithSkipper elastic#133223
  Adding support for index.number_of_replicas to data stream settings (elastic#132748)
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedDocValuesSingleUniqueValue elastic#133221
  Fix VectorSimilarityFunctionsIT (elastic#133206)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupBySubset elastic#133220
  Increase the number of FORK branches in ForkGenerator (elastic#132019)
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=STORED]} elastic#133218
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=DOC_VALUES]} elastic#133217
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=NONE]} elastic#133216
  Set default processor allocation for test clusters (elastic#133204)
  Add mapper for exponential histograms (elastic#132493)
  Fix offset handling in Murmur3Hasher (elastic#133193)
  unmute testDoesNotResolveClosedIndex (elastic#133115)
  Fix an AWS SDK v2 release note (elastic#133155)
  Limit the depth of a filter (elastic#133113)
  ...
szybia added a commit to szybia/elasticsearch that referenced this pull request Aug 20, 2025
@szybia
Merge remote-tracking branch 'upstream/main' into invalid-grok-500
e204957 
* upstream/main: (58 commits)
  Fixing flaky LoggedExec (tests) (elastic#133215)
  CPS search should not use `skip_unavailable` (elastic#132927)
  Don't fail search if bottom doc can't be formatted (elastic#133188)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupByNothing elastic#133225
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedSetDocValuesWithSkipperSmall elastic#133224
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedNumberMergeAwayAllValuesWithSkipper elastic#133223
  Adding support for index.number_of_replicas to data stream settings (elastic#132748)
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedDocValuesSingleUniqueValue elastic#133221
  Fix VectorSimilarityFunctionsIT (elastic#133206)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupBySubset elastic#133220
  Increase the number of FORK branches in ForkGenerator (elastic#132019)
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=STORED]} elastic#133218
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=DOC_VALUES]} elastic#133217
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=NONE]} elastic#133216
  Set default processor allocation for test clusters (elastic#133204)
  Add mapper for exponential histograms (elastic#132493)
  Fix offset handling in Murmur3Hasher (elastic#133193)
  unmute testDoesNotResolveClosedIndex (elastic#133115)
  Fix an AWS SDK v2 release note (elastic#133155)
  Limit the depth of a filter (elastic#133113)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL :Search Relevance/ES|QL Search functionality in ES|QL :SearchOrg/Relevance Label for the Search (solution/org) Relevance team Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search - Relevance The Search organization Search Relevance team Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:SearchOrg Meta label for the Search Org (Enterprise Search) >test Issues or PRs that are addressing/adding tests v9.2.0

3 participants

@ioanatia @elasticsearchmachine @tteofili