[ML] Add ContextualAI inference service

[pedrocassalpacheco]

Add ContextualAI Rerank Service Implementation

Overview

This PR adds a complete implementation of ContextualAI rerank service integration to Elasticsearch's inference plugin, following the established patterns used by other inference services (OpenAI, Cohere, etc.).

What's Implemented

• Complete ContextualAI Service: Full service implementation with rerank task support
• Service Registration: Integrated with InferencePlugin to register the service
• Request/Response Handling: Proper HTTP client integration with ContextualAI API
• Configuration Management: Service settings, task settings, and secret handling
• Rate Limiting: Integrated with Elasticsearch's rate limiting infrastructure
• Instruction Support: Added instruction parameter for guiding rerank behavior

Files Added/Modified

• ContextualAiService.java - Main service implementation
• ContextualAiRerankModel.java - Model representation
• ContextualAiRerankServiceSettings.java - Service configuration
• ContextualAiRerankTaskSettings.java - Task-specific settings with instruction support
• ContextualAiActionCreator.java - Action creation logic
• ContextualAiRerankRequest.java - HTTP request handling with debug logging
• ContextualAiRerankRequestEntity.java - JSON serialization with proper field ordering
• ContextualAiRerankResponseEntity.java - Response parsing
• InferencePlugin.java - Service registration

Lastest commit a818ac3

• This commit has a slightly simpler implementation to support only reranking models.
• It eliminates some of the complexities of supporting different models, but it works for an initial deployment.

Service registration

{
  "service": "contextualai",
  "service_settings": {
    "model_id": "ctxl-rerank-v2-instruct-multilingual",
    "url": "https://api.contextual.ai/v1/rerank",
    "api_key": "your contextual key"
  }
}

Inference test call

{
  "query": "What is the current enterprise pricing for the RTX 5090 GPU for bulk orders?",
  "input": [
    "Following detailed cost analysis and market research, we have implemented the following changes: AI training clusters will see a 15% uplift in pricing effective Q1 2025.",
    "NVIDIA Enterprise Sales Portal: RTX 5090 GPU bulk pricing for enterprise clients is set at $8,500 per unit as of January 2025",
    "TechAnalytics Research Group: Market analysis indicates stable pricing for RTX 5090 GPUs in Q1 2025, with minor fluctuations expected due to supply chain factors.",
    "I really like popcorn"
  ],
   "task_settings": {
    "instruction": "Prioritize internal sales documents over market analysis reports. More recent documents should be weighted higher. Enterprise portal content supersedes distributor communications."
  }
}

Response

{
    "rerank": [
        {
            "index": 1,
            "relevance_score": 0.9999927
        },
        {
            "index": 2,
            "relevance_score": 0.25158092
        },
        {
            "index": 0,
            "relevance_score": 0.00323565
        },
        {
            "index": 3,
            "relevance_score": 2.4E-7
        }
    ]
}

[cla-checker-service]

💚 CLA has been signed

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[pedrocassalpacheco]

LGTM

[davidkyle]

@elasticmachine test this please

[davidkyle]

Looks great thanks @pedrocassalpacheco

Please add tests for the new classes added under this folder - https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services

You can run the tests from your terminal with this command: ./gradlew :x-pack:plugin:inference:test

[davidkyle]

@elasticmachine test this please

[pedrocassalpacheco]

@davidkyle - thank you for your support. Happy to make any further changes if necessary.

[davidkyle]

@elasticmachine test this please

[davidkyle]

@elasticmachine test this please

[davidkyle]

LGTM