Deprecate certificate-based remote cluster security model

[slobodanadamovic]

Today, Elasticsearch supports two models to establish secure connections
and trust between two Elasticsearch clusters:

• API key based security model
• Certificate based security model

This PR deprecates the Certificate based security model in favour of API key based security model.
The API key based security model is preferred way to configure remote clusters,
as it allows to follow security best practices when setting up remote cluster connections
and defining fine-grained access control.

Users are encouraged to migrate remote clusters from certificate to API key authentication.

---

Deprecation details:

The deprecation warning message is logged only at runtime, when a remote connection is first opened. This is in order to avoid overloading the deprecation logs. The logs are deduplicated by passing a key per remote cluster alias (remote_cluster_certificate_access-<remote-cluster-alias>). Worth noting is that with this approach, the warning headers will not be returned to clients that execute a cross cluster search requests (e.g. POST /my_remote_cluster:*/_search). I think this is justifiable, given that end-clients are not the ones interested in such deprecations, but rather administrators of the cluster.

The deprecation message will only be logged on local cluster. This is because there is no way to detect if an inbound connection on a remote cluster is originating from a cluster using certificate based auth. As a consequence, this means that there may be some old local clusters connecting to a new remote clusters using deprecated model, and there would be no deprecations logged at all.

There are no particular settings to be deprecated, because the certificate based model shares the same transport protocol/settings as the intra-cluster node-to-node communication. Hence, we might log a deprecation message for remote connections that are simply missing credentials (cluster.remote.ALIAS.credentials). Edit: This is addressed by checking if the remote connection is using REMOTE_CLUSTER_PROFILE. The remote cluster profile is used only for the new port and transport connections that use new API key based security model.

Note: If the approach makes sense, I will add more test coverage and update the docs.

[elasticsearchmachine]

Hi @slobodanadamovic, I've created a changelog YAML for you. Note that since this PR is labelled >deprecation, you need to update the changelog YAML to fill out the extended information sections.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[bytebilly]

The approach LGTM 👍
for the code changes, I'll defer to engineers :)

[slobodanadamovic]

Linking here docs preview pages:

• remote-clusters-cert.html
• remote-clusters.html
• esql-cross-clusters.html

[n1v0lg]

Thanks for working on this! Briefly discussed on Slack: we also need to cover proxy connections and add a test for it. Few other nit comments, I'll take a quick look again once the changes are in.

[n1v0lg]

Wonder if we want to make this a stronger statement by adding something like "since support for the certificate-based model will be removed in a future version."

[slobodanadamovic]

I've adjusted the changelog to state that it will be removed in future. Let me know if that makes sense: 71fd150

[n1v0lg]

I don't think we should tackle it in this PR, but cross cluster replication docs make a bunch of implicit references to the certificate-based model, e.g.,:

https://www.elastic.co/guide/en/elasticsearch/reference/current/_configure_privileges_for_cross_cluster_replication_2.html

We probably want a follow up of some sorts to update the docs to focus on RCS 2.0 instead, and mention that RCS 1.0 is deprecated.

[slobodanadamovic]

For sure. There is a plan to do that with the help of docs team.

[slobodanadamovic]

Thanks for working on this! Briefly discussed on Slack: we also need to cover proxy connections and add a test for it. Few other nit comments, I'll take a quick look again once the changes are in.

Thanks for catching this! 👍 Please have another look. I've added deprecation logging when a node is connected using proxy mode. Also wrote an integration test which covers both proxy and sniff modes.

[n1v0lg]

LGTM! Neat integ test 💯

[n1v0lg]

very neat!