[ML] Adding headers support for OpenAI chat completion

[jonathan-buttner]

This PR adds support for including custom headers in the chat completion and completion openai request. We'll need the support for other openai task types but I'll do that in a follow up PR to keep this one small.

The headers are stored in the task_settings.

This PR also introduce a new type for the _inference/_services configuration results called map to represent the headers field.

Example create inference API request

PUT _inference/chat_completion/openai
{
    "service": "openai",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "o4-mini"
    },
    "task_settings": {
        "headers": {
            "OpenAI-Organization": "org id",
            "x-request-id": "jon-test"
        }
    }
}

OpenAI configuration

{
    "service": "openai",
    "name": "OpenAI",
    "task_types": [
        "text_embedding",
        "completion",
        "chat_completion"
    ],
    "configurations": {
        "headers": { <---------- New field
            "description": "Custom headers to include in the requests to OpenAI.",
            "label": "Custom Headers",
            "required": false,
            "sensitive": false,
            "updatable": true,
            "type": "map", <-------- New type
            "supported_task_types": [
                "completion",
                "chat_completion"
            ]
        },
        "api_key": {
            "description": "The OpenAI API authentication key. For more details about generating OpenAI API keys, refer to the https://platform.openai.com/account/api-keys.",
            "label": "API Key",
            "required": true,
            "sensitive": true,
            "updatable": true,
            "type": "str",
            "supported_task_types": [
                "text_embedding",
                "completion",
                "chat_completion"
            ]
        },
        "organization_id": {
            "description": "The unique identifier of your organization.",
            "label": "Organization ID",
            "required": false,
            "sensitive": false,
            "updatable": false,
            "type": "str",
            "supported_task_types": [
                "text_embedding",
                "completion",
                "chat_completion"
            ]
        },
        "rate_limit.requests_per_minute": {
            "description": "Default number of requests allowed per minute. For text_embedding is 3000. For completion is 500.",
            "label": "Rate Limit",
            "required": false,
            "sensitive": false,
            "updatable": false,
            "type": "int",
            "supported_task_types": [
                "text_embedding",
                "completion",
                "chat_completion"
            ]
        },
        "model_id": {
            "description": "The name of the model to use for the inference task.",
            "label": "Model ID",
            "required": true,
            "sensitive": false,
            "updatable": false,
            "type": "str",
            "supported_task_types": [
                "text_embedding",
                "completion",
                "chat_completion"
            ]
        },
        "url": {
            "description": "The absolute URL of the external service to send requests to.",
            "label": "URL",
            "required": false,
            "sensitive": false,
            "updatable": false,
            "type": "str",
            "supported_task_types": [
                "text_embedding",
                "completion",
                "chat_completion"
            ]
        },
        "dimensions": {
            "description": "The number of dimensions the resulting embeddings should have. For more information refer to https://platform.openai.com/docs/api-reference/embeddings/create#embeddings-create-dimensions.",
            "label": "Dimensions",
            "required": false,
            "sensitive": false,
            "updatable": false,
            "type": "int",
            "supported_task_types": [
                "text_embedding"
            ]
        }
    }
}

[elasticsearchmachine]

Hi @jonathan-buttner, I've created a changelog YAML for you.

[jonathan-buttner]

Randomly found that this was testing the wrong class so switch it to the appropriate one.

[jonathan-buttner]

Since we need to add the headers to each task type I'll move this functionality to an abstract class in a later PR.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[DonalEvans]

Minor typo, should be "the returned Map is immutable"

[DonalEvans]

It would be good to also have a test for the case where the headers map is null, since that's new functionality in this PR.