Adding support for binary embedding type to Cohere service embedding type

[ymao1]

Towards #111747

Summary

Adds support for binary embedding type in the Cohere text_embedding task.

Usage

# Create a Cohere binary inference endpoint
PUT /_inference/text_embedding/cohere_embeddings_bit
{
    "service": "cohere",
    "service_settings": {
        "api_key": <apiKey>,
        "model_id": "embed-english-v3.0",
        "embedding_type": "bit"
    }
}

# Perform an inference task
POST /_inference/text_embedding/cohere_embeddings_bit
{
    "input": "hello",
    "task_settings": {
        "input_type": "ingest"
    }
}

# Response
{
    "text_embedding_bits": [
        {
            "embedding": [
                -55,
                74,
                101,
                67,
                83,
                1,
                53,
                -101,
                -71,
                -98,
                -116,
                -99,
                80,
                -49,
                65,
                .
                .
                .
            ]
        }
    ]
}

Notes

Requesting a binary embedding from Cohere returns an array of binary embeddings encoded as bytes with int8 precision. Since this aligns with what is expected as input when you specify a dense_vector mapping with element_type: bit, we do not perform any bit unpacking on the Cohere response and handle the bytes as-is.

[davidkyle]

Nice!

This looks good to merge, please make it a non draft PR when you are ready.

If it is simple to add in this PR then Jina AI also have binary embeddings. They have a good blog at https://jina.ai/news/binary-embeddings-all-the-ai-3125-of-the-fat/

[elasticsearchmachine]

Hi @ymao1, I've created a changelog YAML for you.

[ymao1]

If it is simple to add in this PR then Jina AI also have binary embeddings.

I will do this in a follow-up PR!

[github-actions]

It looks like this PR modifies one or more .asciidoc files. These files are being migrated to Markdown, and any changes merged now will be lost. See the migration guide for details.

[ymao1]

Reverted the docs change due to the docs freeze, looks like I'll have to create a PR to 8.x for the asciidocs change and then an issue in https://github.com/elastic/docs-content/issues to get the docs update to 9.0

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

LGTM

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 120751

[ymao1]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

[ymao1]

Documentation request made in elastic/docs-content#310