Release DiskBBQ(bbq_disk) index type for dense_vector fields

[benwtrent]

This releases a new index type called DiskBBQ (configuration bbq_disk).

DiskBBQ is a cluster based format that provides:

• faster and cheaper indexing than HNSW
• Better behavior in lower memory environments (degrades linearly, not exponentially)
• Is near HNSW for QPS when the index is in memory

Current restrictions that will be able to be removed in the future:

• only floating point values are allowed currently
• quantization is only to a single bit, so not recommended for low dimensionality vectors
• all other restrictions for hnsw

To utilize the format, its just like any other:

PUT vectors {
  "mappings": {
    "properties": {
      "vector": {"type": "dense_vector", "index_options": {"type": "disk_bbq"}
    }
  }
}

Querying is just like any other field.

POST vectors/_search
{
    "query": {
        "knn": {
            "field": "vector",
            "query_vector": <vector,
            "k": 3
        }
    }
}

num_candidates can be used for tuning approximate nature of the search. Or, more granular control can be provided by setting visit_percentage directly.

[elasticsearchmachine]

Hi @benwtrent, I've created a changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[github-actions]

🔍 Preview links for changed docs

• docs/reference/elasticsearch/mapping-reference/dense-vector.md
• docs/reference/elasticsearch/rest-apis/retrievers/knn-retriever.md
• docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[john-wagster]

lgtm. I had a few thoughts while reviewing that since the yaml tests weren't touched at all I wonder if they need to be updated. It doesn't look like anything is missing to me but things like 20_knn_retriever.yml which is gated with mapper.bbq_disk_support is what I was thinking about. Probably fine but figured I'd mention it since it passed through my head as I was reviewing.

[john-wagster]

failing validation because the spacing is different here (5 spaces instead of 4) beginning with Current restrictions

[john-wagster]

I think these need to be "----" instead of "```"; I could be wrong. But "----" passes validation

[tteofili]

LGTM

[jonasjancarik]

To utilize the format, its just like any other:

PUT vectors {
  "mappings": {
    "properties": {
      "vector": {"type": "dense_vector", "index_options": {"type": "disk_bbq"}
    }
  }
}

That example is also in the release notes, but it should actually be "type": "bbq_disk" I believe. The docs are correct (https://www.elastic.co/docs/reference/elasticsearch/index-settings/bbq#bbq-disk)