[Vector Search] BugFix wrong vector docvalue_fields

[weizijun]

The denormalizedCosineFloatVectorValues class is missing the ordToDoc method, which can lead to incorrect ordering and potentially incorrect values. Reproduce the error：

PUT test_cosine
{
  "mappings": {
    "properties": {
      "foo" : {
        "type" : "keyword"
      },
      "vec" : {
        "type": "dense_vector",
        "similarity": "cosine"
      }
    }
  }
}

PUT test_cosine/_doc/1
{
  "foo" : "bar"
}

PUT test_cosine/_doc/2
{
  "foo" : "bar"
}

PUT test_cosine/_doc/3
{
  "vec" : [1,2]
}

POST test_cosine/_refresh

PUT test_cosine/_doc/4
{
  "foo" : "bar"
}

POST test_cosine/_forcemerge?max_num_segments=1

GET test_cosine/_search
{
  "docvalue_fields": ["vec"]
}

the wrong value is :

"vec": [
  [
    0.4472136,
    0.8944272
  ]
]

[weizijun]

Hi, @benwtrent , can you help to review this PR?

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[iverase]

@elasticmachine test this please

[iverase]

@elasticmachine test this please

[iverase]

@elasticmachine test this please

[iverase]

Your new test fails in a mixed cluster setting, we need to skip those situations:

./gradlew ":qa:mixed-cluster:v9.2.2#mixedClusterTest" -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.vectors/200_dense_vector_docvalue_fields/Dense vector cosine docvalue_fields with sparse documents after force merge}" -Dtests.seed=2FFDEDC536B94BAE -Dtests.bwc=true -Dtests.locale=no -Dtests.timezone=Asia/Kolkata -Druntime.java=25

[weizijun]

Your new test fails in a mixed cluster setting, we need to skip those situations:

./gradlew ":qa:mixed-cluster:v9.2.2#mixedClusterTest" -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.vectors/200_dense_vector_docvalue_fields/Dense vector cosine docvalue_fields with sparse documents after force merge}" -Dtests.seed=2FFDEDC536B94BAE -Dtests.bwc=true -Dtests.locale=no -Dtests.timezone=Asia/Kolkata -Druntime.java=25

I fixed it by add a search capability flag.

[iverase]

@elasticmachine test this please

[iverase]

@elasticmachine test this please

[iverase]

@elasticmachine test this please

[benwtrent]

Great find @weizijun !

Thank you for guiding it through @iverase

[iverase]

I fixed it by add a search capability flag.

@weizijun I hope you don't mind but I have moved it to a node test feature. Search capabilities have an operational cost and in this case we just want to skip some test configurations.

[weizijun]

@weizijun I hope you don't mind by I have moved it to a node test feature. Search capabilities have an operational cost and in this case we just want to skip some test configurations.

It's ok, Thank you!

[iverase]

Thanks @weizijun!

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	9.2	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.1	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 137862

[iverase]

I have backported manually to 9.2 and 9.1. It seems this is not needed in 8.19.