feat: VoyageAI integration

[fzowl]

VoyageAI embeddings and rerank model integration through the API (https://docs.voyageai.com/reference/embeddings-api and https://docs.voyageai.com/reference/reranker-api)

[elasticsearchmachine]

@fzowl please enable the option "Allow edits and access to secrets by maintainers" on your PR. For more information, see the documentation.

[fzowl]

@davidkyle, can you please take a look? (sorry, this PR became huge with all the integrations and the tests)

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

@elasticmachine test this please

[jonathan-buttner]

Hmm I think spotless is complaining, running spotless like this might help:

./gradlew :x-pack:plugin:core:spotlessApply :x-pack:plugin:core:precommit :x-pack:plugin:ml:spotlessApply :x-pack:plugin:ml:precommit :x-pack:plugin:inference:spotlessApply :x-pack:plugin:inference:precommit :x-pack:plugin:ml:qa:native-multi-node-tests:precommit :server:precommit

[fzowl]

@jonathan-buttner Thanks! This solved the problem for me: ./gradlew :x-pack:plugin:inference:qa:inference-service-tests:spotlessApply

[jonathan-buttner]

@elasticmachine test this please

[fzowl]

@jonathan-buttner can you please help me here? I found that i can reproduce the test with ./gradlew ":rest-api-spec:yamlRestCompatTest" --tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT" -Dtests.method="test {yaml=indices.create/21_synthetic_source_stored/index param - field ordering}" -Dtests.seed=29856D0D59D74338 -Dtests.locale=csw -Dtests.timezone=US/Pacific -Druntime.java=23 and i could reproduce....but honestly, i don't know why it fails.

Can you please point me to the right direction to resolve this? Also, I gave you write permission so please feel free to make any adjustment required. Thank you!

[jonathan-buttner]

@elasticmachine merge upstream

[jonathan-buttner]

@jonathan-buttner can you please help me here?

Yeah that's probably a flaky test. Try merging in main and I'll run the tests again.

[jonathan-buttner]

Also, I gave you write permission so please feel free to make any adjustment required. Thank you!

Oh just saw that, thanks! Let me see if I can merge upstream for you

[jonathan-buttner]

@fzowl hmm, I'm having trouble pushing directly to the upstream. Normally I can gh pr checkout 122134 and then push directly but doesn't seem like that's working:

❯ git push
fatal: The upstream branch of your current branch does not match
the name of your current branch.  To push to the upstream branch
on the remote, use

    git push upstream HEAD:refs/pull/122134/head

To push to the branch of the same name on the remote, use

    git push upstream HEAD

To avoid automatically configuring an upstream branch when its name
won't match the local branch, see option 'simple' of branch.autoSetupMerge
in 'git help config'.

I merged main, and had to resolve the conflicts in the TransportVersions file. Looks like the number changed so we needed to bump the public static final TransportVersion VOYAGE_AI_INTEGRATION_ADDED_BACKPORT_8_X = def(8_841_0_05); note the 8_841_0_05 instead of 8_840_0_05.

Could you try merging upstream?

[jonathan-buttner]

Never mind, I think I was able to do it!

[jonathan-buttner]

@elasticmachine test this please

[fzowl]

I merged main, and had to resolve the conflicts in the TransportVersions file. Looks like the number changed so we needed to bump the public static final TransportVersion VOYAGE_AI_INTEGRATION_ADDED_BACKPORT_8_X = def(8_841_0_05); note the 8_841_0_05 instead of 8_840_0_05.

Yeah, i noticed that a new minor version is created very frequently (for every merged PR?), that's why i wanted to modify TransportVersions right before you merge my PR: #122134

[jonathan-buttner]

@elasticmachine test this please

[jonathan-buttner]

@elasticmachine test this please

[fzowl]

@jonathan-buttner This looks good now!

[fzowl]

@jonathan-buttner Thank you very much for your help and support!

[jonathan-buttner]

Thanks for all the work to get this implemented! Glad I could help.

[jonathan-buttner]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation