Enable bfloat16 and on-disk rescoring for dense vectors

[thecoop]

New options have been added to the dense_vector field type.

The first is support for storing vectors in bfloat16 format. This is a floating-point format that utilises two bytes per value rather than four, halving the storage space required compared to element_type: float. This can be specified with element_type: bfloat16 when creating the index, for all dense_vector indexing types:

PUT vectors
{
  "mappings": {
    "properties": {
      "vector": {
        "type": "dense_vector",
        "element_type": "bfloat16",
        "index_options": {"type": "bbq_hnsw"}
      }
    }
  }
}

Float values are automatically rounded to 2-bytes when writing to disk, so this format can be used with original source vectors at 2- or 4-byte precision. BFloat16 values are zero-expanded back to 4-byte floats when read into memory. Using bfloat16 will cause a loss of precision compared to the original vector values, as well as a small performance hit due to converting between bfloat16 and float when reading and writing vectors; however this may be counterbalanced by a corresponding decrease in I/O, depending on your workload.

The second option is to utilise on-disk rescoring. When rescoring vectors during kNN searches, the raw vectors are read into memory. When the vector data is larger than the amount of available RAM, this might cause the OS to evict some in-memory pages that then need to be paged back in immediately afterwards. This can cause a significant slowdown in search speed. Utilising on-disk rescoring causes rescoring to use raw vector data on-disk during rescoring, and to not read it into memory first. This may significantly increase search performance in such low-memory situations.
This is specified as an index option for HNSW vector index types:

PUT vectors
{
  "mappings": {
    "properties": {
      "vector": {
        "type": "dense_vector",
        "index_options": {"type": "bbq_hnsw", "on_disk_rescore": true}
      }
    }
  }
}

bfloat16 support is GA, on-disk rescoring is preview functionality.

Doc changes: elastic/docs-content#3847
Spec changes: elastic/elasticsearch-specification#5728

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @thecoop, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/elasticsearch/mapping-reference/dense-vector.md

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Hi @thecoop, I've updated the changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

[tteofili]

LGTM

[shainaraskas]

some suggestions for signaling availability

[shainaraskas]

added some comments to the changelog entry while I was answering your question :)

[shainaraskas]

added some comments to the changelog entry while I was answering your question :)

[jimczi]

Let's split this in two please, one PR to enable bfloat16 and another one for rescoring dense vectors.

[jimczi]

I understand that we have a single feature flag here so we cannot split. I am approving to unblock but let's at least have a separate change log that we can link with the appropriate issue.