[Elasticsearch]: Authentication fails when password contains YAML special characters (e.g. double quotes) #18434

Integration Name

Elasticsearch [elasticsearch]

Dataset Name

All metrics datasets (cluster_stats, node_stats, node, index, index_recovery, index_summary, ccr, enrich, ingest_pipeline, ml_job, pending_tasks, shard)

Integration Version

1.20.1

Agent Version

8.19.1

Agent Output Type

elasticsearch

Elasticsearch Version

8.19.1

OS Version and Architecture

Linux (x86_64)

Software/API Version

N/A

Error Message

The elasticsearch/metrics input enters a Degraded state with HTTP 401 errors when the configured password contains YAML special characters such as double quotes ("):

Degraded

Error fetching data for metricset elasticsearch.cluster_stats: error determining if
connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized

The same error pattern appears across all metrics datasets that share the same template structure.

Event Original

N/A — this is an authentication/configuration issue, not a pipeline/parsing issue.

What did you do?

Configured the Elasticsearch integration's metrics input (elasticsearch/metrics) with a username and password where the password value contains double-quote characters, for example:

password: "mysecret"

(i.e. the literal password string includes the " characters as part of the value, not as YAML delimiters)

What did you see?

All metrics datasets immediately enter a Degraded state with 401 Unauthorized responses from Elasticsearch.

Inspecting the compiled agent configuration (components-actual.yaml) shows the password is stored without the surrounding quote characters:

# With a password value of: "mysecret"
password:
  kind:
    stringvalue: mysecret   # <-- double quotes stripped by YAML parser

This happens because the Handlebars template renders the password directly as an unquoted YAML scalar:

# stream.yml.hbs (all metrics datasets)
{{#if password}}
password: {{password}}
{{/if}}

When {{password}} expands to "mysecret", the rendered YAML line becomes:

password: "mysecret"

YAML interprets the double quotes as string delimiters, so the parsed value is mysecret — the quotes are silently dropped. The agent then authenticates with the wrong password.

Verification with curl confirms the distinction:

# Fails (quotes stripped by shell — same as what the agent experiences):
curl -k -u monitoring_user:"mysecret" -X GET 'https://localhost:9200/_cluster/stats?pretty'
# {"type":"security_exception","reason":"unable to authenticate user..."}

# Succeeds (quotes preserved via single-quoting):
curl -k -u 'monitoring_user:"mysecret"' -X GET 'https://localhost:9200/_cluster/stats?pretty'

Logging in via Kibana UI with the same password works correctly because the UI handles the value as a raw string.

What did you expect to see?

Authentication should succeed regardless of whether the password contains YAML special characters. The compiled agent config should preserve the password value exactly as entered, including any double quotes, backslashes, or other special characters.

Anything else?

Root cause: The .hbs stream templates for every Elasticsearch metrics dataset use password: {{password}} without quoting, so the YAML parser interprets characters like " as structural syntax rather than literal content.

Fix: Replace {{password}} with {{escape_string password}} in all 12 metrics dataset stream templates. The escape_string Handlebars helper (registered in Fleet's template engine) wraps the value in YAML single quotes and escapes any interior single quotes by doubling them, safely preserving all special characters.

For example, a password of "mysecret" would render as:

password: '"mysecret"'

which YAML correctly parses back to the string "mysecret" (with the double quotes intact).

The same pattern is already used correctly in other integrations (e.g. ibmmq, apache_tomcat, websphere_application_server) and with other sensitive fields in cloudflare (escape_string auth_key).

Workaround: Use a password that does not contain YAML special characters, or authenticate via API key instead of username/password.