[ML] CustomService adding template validation prior to request flow

[jonathan-buttner]

This PR addresses the feedback item from the original PR: #127939

This PR adds some additional validation logic prior to the typical test request validation. This addresses the validation error being nested a couple levels deep when the traditional validation is executed.

The error would also be caught in the normal request validation logic but it is nested a few levels. This change performs the initial validation for schema before making an actual request.

Comments:
#127939 (comment)

Example error after this change

{
    "error": {
        "root_cause": [
            {
                "type": "validation_exception",
                "reason": "Validation Failed: 1: Failed to validate model configuration: Found placeholder [${api_keys}] in field [header.Authorization] after replacement call, please check that all templates have a corresponding field definition.;"
            }
        ],
        "type": "validation_exception",
        "reason": "Validation Failed: 1: Failed to validate model configuration: Found placeholder [${api_keys}] in field [header.Authorization] after replacement call, please check that all templates have a corresponding field definition.;"
    },
    "status": 400
}

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

LGTM

Definitly an improvement, this change should make it much easier to configure a custom service

[davidkyle]

Is there a more specific exception to catch, perhaps define a new exception class CustomServiceSubsitutionException extends ElasticsearchException and throw that in ValidatingSubstitutor

[davidkyle]

👍

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.19