Use the new merge executor for intra-merge parallelism

[benwtrent]

Now that we have a nice shiny executor service to serve background merges, let's steal some threads for intra-merge parallelism...

For merge isolated actions, this is an almost linear speed improvement by throwing more threads at the HNSW graph merge. Of course, this uses more resources.

Benchmarks show that since now we restrict the threadpool to be the same as the merging pool, there is no impact on resource utilization when indexing & querying on the same nodes.

However, if there aren't many merges occurring, and the user is force-merging, this sucker is a huge improvement. Maybe we should start giving more threads to merges... ;)

[elasticsearchmachine]

Hi @benwtrent, I've created a changelog YAML for you.

[benwtrent]

Buildkite benchmark this with so-vector please

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 34401b1
• Baseline: efe4d42 (env ID 10572b20-8d09-416e-a619-1749393cf5b6)
• Contender: 34401b1 (env ID 754c53f3-6ad1-4150-9d37-570705753d6c)
• Benchmark results

This build ran two so-vector benchmarks to evaluate performance impact of this PR.

History

[benwtrent]

Benchmarks show nice improvement:

Need to benchmark with parallel search

[benwtrent]

OK, benchmarked again, just on OpenAI dataset. There seems to be pretty much no impact on things we are trying to measure (parallel search & indexing). I think this is because we are simply using the same threads as the merging thread pool, which allows things to be throttled and partitioned just like a regular merge. So, if just a single merge was happening, it would use all the merge threads.

candidate:
https://gist.github.com/benwtrent/5ba71f2017257779475b58c34336d9b3#file-multi_candidate-txt

baseline:
https://gist.githubusercontent.com/benwtrent/0c0c1e16bb82acab558c98e1863421e1/raw/7ce4f80fa204c5ef9625d86e031adb4eaa527cd8/baseline.txt

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[iverase]

This PR gives a good overview oh how we hook index settings to vector formats. It looks good to me, I am just wondering if we need some test to check that the setting is propagated correctly to the format.

[iverase]

Any thoughts on making true the default?

[benwtrent]

@iverase we could. Let me see if distrib indexing has any opinion here.

[benwtrent]

@iverase I am thinking we first release this setting as "preview" and default its value if the build is "snapshot" or not.

[iverase]

sounds like a good plan to me.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/elasticsearch/index-settings/merge.md

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[iverase]

LGTM