[Elasticsearch][Ingest pipeline] Stop truncating the elasticsearch server log messages

[consulthys]

Proposed commit message

This PR fixes the ingest pipeline called logs-elasticsearch.server-<version>-pipeline-json that parses elasticsearch server logs.

The pipeline was inherited from the Filebeat elasticsearch module and hasn't changed in several years. The main issue is that the pipeline makes the assumption that if the message field value starts with square brackets (e.g. [xyz] some log message), then xyz is considered to be an index name (indexed in the elasticsearch.index.name field) and the message is truncated to only what comes after the square brackets (i.e. some log message). This assumption might have been true at some point in the past, but isn't the case anymore, i.e. the square brackets can contain literally anything, such as component names, class names, etc. Truncating the message field breaks downstream processes that expect to find the full log message in that field.

For instance, when applied on the following log message

[co.elastic.elasticsearch.metering.sampling.SampledStorageMetricsProvider] is not ready for collect yet

the truncated message field will only contain is not ready for collect yet, which now lacks context and is unsuable.

As it is not easy to find out all (internal and external) downstream processes that rely on the index name to be extracted from the log message, we need to keep extracting whatever is in the square brackets, but without truncating the message field. This PR suggests a non-breaking change that will keep extracting the index name (if it doesn't already exist in the document), while leaving the message field alone.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have added an entry to my package's changelog.yml file.

How to test this PR locally

1. Install the elasticsearch integration update
2. Run the following ingest pipeline simulation in Dev Tools and make sure that the message field is unaltered

POST _ingest/pipeline/logs-elasticsearch.server-1.17.2-pipeline-json/_simulate
{
  "docs": [
    {
      "_source": {
        "@timestamp": "2025-01-22T15:50:04.517Z",
        "log.level": "INFO",
        "message": "[co.elastic.elasticsearch.metering.sampling.SampledStorageMetricsProvider] is not ready for collect yet",
        "ecs.version": "1.2.0",
        "service.name": "ES_ECS",
        "event.dataset": "elasticsearch.server",
        "process.thread.name": "elasticsearch[es-es-search-c886b6975-9tsbz][metering_reporter][T#1]",
        "log.logger": "co.elastic.elasticsearch.metering.usagereports.UsageReportCollector",
        "elasticsearch.cluster.uuid": "vaYFhXk-Q0WFOdSjyaylnA",
        "elasticsearch.node.id": "fnqA_AXpTFm6uZdtXDNIPA",
        "elasticsearch.node.name": "es-es-search-c886b6975-9tsbz",
        "elasticsearch.cluster.name": "es"
      }
    }
  ]
}

Response:

{
  "docs": [
    {
      "doc": {
        "_source": {
          "elasticsearch": {
            "index": {
>>>>          "name": "co.elastic.elasticsearch.metering.sampling.SampledStorageMetricsProvider"
            }
          },
          ...
>>>>      "message": "[co.elastic.elasticsearch.metering.sampling.SampledStorageMetricsProvider] is not ready for collect yet",
          ...
        }
      }
    }
  ]
}

If the elasticsearch.index.name field is already present in the log document, then it will not be overridden

POST _ingest/pipeline/logs-elasticsearch.server-1.17.2-pipeline-json/_simulate
{
  "docs": [
    {
      "_source": {
        "@timestamp": "2025-01-22T15:50:04.517Z",
        "log.level": "INFO",
        "message": "[co.elastic.elasticsearch.metering.sampling.SampledStorageMetricsProvider] is not ready for collect yet",
        "ecs.version": "1.2.0",
        "service.name": "ES_ECS",
        "event.dataset": "elasticsearch.server",
        "process.thread.name": "elasticsearch[es-es-search-c886b6975-9tsbz][metering_reporter][T#1]",
        "log.logger": "co.elastic.elasticsearch.metering.usagereports.UsageReportCollector",
        "elasticsearch.cluster.uuid": "vaYFhXk-Q0WFOdSjyaylnA",
        "elasticsearch.node.id": "fnqA_AXpTFm6uZdtXDNIPA",
        "elasticsearch.node.name": "es-es-search-c886b6975-9tsbz",
        "elasticsearch.cluster.name": "es",
>>>     "elasticsearch.index.name": "index-123"
      }
    }
  ]
}

Response:

{
  "docs": [
    {
      "doc": {
        "_source": {
          "elasticsearch": {
            "index": {
>>>>          "name": "index-123"
            }
          },
          ...
>>>>      "message": "[co.elastic.elasticsearch.metering.sampling.SampledStorageMetricsProvider] is not ready for collect yet",
          ...
        }
      }
    }
  ]
}

Related issues

Closes #12501

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b479c12

History

• 💔 Build #22457 failed b6a2b95
• 💚 Build #22418 succeeded 9eb9015
• 💔 Build #22415 failed ee0b087

cc @consulthys

[pickypg]

LGTM

[elastic-vault-github-plugin-prod]

Package elasticsearch - 1.17.2 containing this change is available at https://epr.elastic.co/package/elasticsearch/1.17.2/