Integrate watsonx for re-ranking task

[saikatsarkar056]

These are related to this Issue.

This PR involves the integation of IBM watsonx re-ranking. The changes leverage the reranker model and the text reranker API provided by watsonx.ai to order a collection of passages based on their relevance to a given query, ranking them from most to least likely to provide an answer.

Screen.Recording.2025-01-27.at.5.42.57.PM.mov

PUT _inference/rerank/ibm_watsonx_rerank
{
    "service": "watsonxai",
    "service_settings": {
        "api_key": <API_KEY>,
        "url": "us-south.ml.cloud.ibm.com",
        "model_id": "cross-encoder/ms-marco-minilm-l-12-v2",
        "project_id": <PROJECT_ID>,
        "api_version": "2024-05-02"
    }
}

POST _inference/rerank/ibm_watsonx_rerank
{
  "query": "database",
  "input": [
  "Greenland is the world's largest island and is known for its vast tundra and immense glaciers. It is an autonomous territory of Denmark.",
  "Google is a multinational technology company specializing in internet-related services such as search engines, online advertising, and cloud computing.",
  "Harry Potter is a fictional wizard created by J.K. Rowling, featured in a series of novels and films. The story revolves around his journey to defeat the dark wizard Voldemort.",
  "Canada is the second-largest country in the world by land area, known for its natural beauty, multicultural population, and strong economy.",
  "MySQL is a popular open-source relational database management system. It is widely used in web applications and supports structured query language (SQL) for database operations.",
  "Solar panels are a sustainable energy solution that convert sunlight into electricity. They are increasingly used for residential and commercial purposes to reduce carbon footprints."
],
  "task_settings": {
        "truncate_input_tokens": 50,
        "return_documents": true,
        "top_n": 2
    }
}

Response

{
  "rerank": [
    {
      "index": 4,
      "relevance_score": -1.889183,
      "text": "MySQL is a popular open-source relational database management system. It is widely used in web applications and supports structured query language (SQL) for database operations."
    },
    {
      "index": 2,
      "relevance_score": -3.2549715,
      "text": "Harry Potter is a fictional wizard created by J.K. Rowling, featured in a series of novels and films. The story revolves around his journey to defeat the dark wizard Voldemort."
    }
  ]
}

[elasticsearchmachine]

Pinging @elastic/search-experiences-team (Team:Search - Experiences)

[elasticsearchmachine]

Pinging @elastic/search-eng (Team:SearchOrg)

[jonathan-buttner]

Looking good, left a few comments.

[thecoop]

This PR was created before 9.0 and 8.18 were branched. Where does this need to be backported to?

[saikatsarkar056]

@thecoop
This should be backported to 8.18 and 9.0

[saikatsarkar056]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

[serenachou]

@saikatsarkar056 does this mean it made it into 8.18/9.0 after all?

[saikatsarkar056]

@serenachou This means these changes made it to 8.19 and 9.0. I tested reranking in those deployments.

[saikatsarkar056]

💚 All backports created successfully

Status	Branch	Result
✅	8.18

Questions ?

Please refer to the Backport tool documentation