[ML] Use internal user for internal inference action

[jonathan-buttner]

When we added the proxy action it caused a bug because we moved the InferenceAction.Request to an internal action instead of monitor(here). There are a few places where we're using the InferenceAction.Request directly. In these situations we need to execute the client using the INFERENCE_ORIGIN to indicate that we're acting as the internal user. Otherwise we'd get an error when a read only user (with only monitor_inference privileges) attempts to make inference requests.

Reproducing the issue

Details

The issue can be reproduced like the following:

PUT _inference/rerank/coherererank
{
    "service": "cohere",
    "service_settings": {
        "api_key": "api_key",
        "model_id": "rerank-v3.5"
    }
}

PUT jon/_doc/1?pretty
{
    "title": "The Terminator",
    "overview": "A cyborg is sent back in time to kill Sarah Connor."
}

PUT jon/_doc/2?pretty
{
    "title": "Terminator 2: Judgment Day",
    "overview": "A cyborg is sent back in time to protect John Connor."
}
PUT jon/_doc/3?pretty
{
    "title": "Terminator Genisys",
    "overview": "A cyborg is sent back in time to protect Sarah Connor."
}

GET jon/_search <-- This should result in a permissions error
{
    "_source": [
        "title"
    ],
    "retriever": {
        "text_similarity_reranker": {
            "retriever": {
                "standard": {
                    "query": {
                        "multi_match": {
                            "fields": [
                                "title",
                                "overview"
                            ],
                            "query": "terminator arnold"
                        }
                    }
                }
            },
            "field": "title",
            "inference_text": "terminator arnold",
            "inference_id": "coherererank"
        }
    }
}

POST /_query?format=txt <--- this should also fail
{
    "query": "FROM jon | KEEP title, overview | SORT title DESC | LIMIT 10 | RERANK \"terminator arnold\" ON title WITH coherererank"
}

Error

{
    "error": {
        "root_cause": [
            {
                "type": "status_exception",
                "reason": "[text_similarity_reranker] search failed - retrievers '[standard]' returned errors. All failures are attached as suppressed exceptions.",
                "suppressed": [
                    {
                        "type": "search_phase_execution_exception",
                        "reason": "Computing updated ranks for results failed",
                        "phase": "rank-feature",
                        "grouped": true,
                        "failed_shards": []
                    }
                ]
            }
        ],
        "type": "status_exception",
        "reason": "[text_similarity_reranker] search failed - retrievers '[standard]' returned errors. All failures are attached as suppressed exceptions.",
        "suppressed": [
            {
                "type": "search_phase_execution_exception",
                "reason": "Computing updated ranks for results failed",
                "phase": "rank-feature",
                "grouped": true,
                "failed_shards": [],
                "caused_by": {
                    "type": "security_exception",
                    "reason": "action [cluster:internal/xpack/inference] is unauthorized for user [test_read_user] with effective roles [test_read], this action is granted by the cluster privileges [manage,all]"
                }
            }
        ]
    },
    "status": 403
}

Testing that it works correctly

To ensure the fix works we can do the following

Details

Create a role with only monitor_inference

POST _security/role/test_read?pretty
{
    "cluster": ["monitor_inference"],
    "indices": [
      {
        "names": [
          "jon*"
        ],
        "privileges": [
          "read"
        ],
        "allow_restricted_indices": false
      }
    ]
}

Create a user that uses the role

POST _security/user/test_read_user?pretty
{
    "password": "password",
    "roles": [
      "test_read"
    ]
}

Perform the request with the test_read_user

GET jon/_search
{
    "_source": [
        "title"
    ],
    "retriever": {
        "text_similarity_reranker": {
            "retriever": {
                "standard": {
                    "query": {
                        "multi_match": {
                            "fields": [
                                "title",
                                "overview"
                            ],
                            "query": "terminator arnold"
                        }
                    }
                }
            },
            "field": "title",
            "inference_text": "terminator arnold",
            "inference_id": "coherererank"
        }
    }
}

POST /_query?format=txt
{
    "query": "FROM jon | KEEP title, overview | SORT title DESC | LIMIT 10 | RERANK \"terminator arnold\" ON title WITH coherererank"
}

[elasticsearchmachine]

Hi @jonathan-buttner, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.0	Commit could not be cherrypicked due to conflicts
❌	8.18	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 128327

[jonathan-buttner]

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.19
✅	8.18

Questions ?

Please refer to the Backport tool documentation