metricbeats: update monitor mappings to include new TBS metrics

[ericywl]

Proposed commit message

Update metricbeat beat module mapping to contain latest apm-server monitoring metrics fields to improve TBS.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Author's Checklist

• Verify mappings are present in the metricbeat-* datastream.

How to test this PR locally

Step 1: Ensure Elasticsearch & Kibana is running

> cd apm-server
> docker-compose up elasticsearch kibana

Step 2: Create APM Server config

Create an apm-server.yml config file:

apm-server:
  host: "127.0.0.1:8200"

output.elasticsearch:
  enabled: true
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

http:
  enabled: true
  host: localhost
  port: 5066

Step 3: Start APM Server

Run APM Server binary directly:

> cd apm-server
> ./apm-server -e -v -c apm-server.yml

Step 4: Configure Metricbeat WITHOUT xpack

Create metricbeat.yml config file:

metricbeat.modules:
- module: beat
  metricsets:
    - state
    - stats
  period: 10s
  hosts: ["http://localhost:5066"]

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

Step 5: Start Metricbeat

Run Metricbeat binary directly:

> cd beats
> make python-env
> source build/python-env/bin/activate

> cd beats/metricbeat
> make mage
> mage build

> sudo chown root modules.d/system.yml
> sudo chown root metricbeat.yml
> sudo ./metricbeat -c metricbeat.yml -e

Step 6: Verify Datastream

Verify fields in datastream metricbeat-*

GET metricbeat-*/_search
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"term": {"metricset.name": "stats"}},
        {"term": {"beat.type": "apm-server"}}
      ]
    }
  },
  "_source": ["beat.stats.apm_server.sampling.tail.storage"],
  "sort": [{"@timestamp": {"order": "desc"}}]
}

"hits": [
  {
    "_index": ".ds-metricbeat-9.3.0-2026.01.13-000001",
    "_id": "o1aSmZoBfa9YIotqCAfv",
    "_score": null,
    "_source": {
      "beat": {
        "stats": {
          "apm_server": {
            "sampling": {
              "tail": {
                "storage": {
                  "disk_total": 994662584320,
                  "disk_usage_threshold_pct": 80.2,
                  "disk_used": 336527384576,
                  "lsm_size": 10553,
                  "storage_limit": 0,
                  "value_log_size": 0
                }
              }
            }
          }
        }
      }
    },
    "sort": [
      1763513067282
    ]
  }
]

Verify mappings for datastream metricbeat-*

GET metricbeat-*/_mapping?filter_path=**.apm_server.**.storage

{
  ".ds-metricbeat-9.3.0-2026.01.14-000001": {
    "mappings": {
      "properties": {
        "beat": {
          "properties": {
            "stats": {
              "properties": {
                "apm_server": {
                  "properties": {
                    "sampling": {
                      "properties": {
                        "tail": {
                          "properties": {
                            "storage": {
                              "properties": {
                                "disk_total": {
                                  "type": "float"
                                },
                                "disk_usage_threshold_pct": {
                                  "type": "float"
                                },
                                "disk_used": {
                                  "type": "float"
                                },
                                "lsm_size": {
                                  "type": "long"
                                },
                                "storage_limit": {
                                  "type": "long"
                                },
                                "value_log_size": {
                                  "type": "long"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Related issues

• Relates monitoring: expose storage limit and disk metrics for TBS monitoring apm-server#15533

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[github-actions]

🔍 Preview links for changed docs

• docs/reference/metricbeat/exported-fields-beat.md

[pickypg]

LGTM

[consulthys]

@ericywl thanks for this enhancement. Just out of curiosity, do you also intend to update the associated dashboards in the beat integration?

[vishaangelova]

docs LGTM

[github-actions]

@Mergifyio backport 9.2 9.3

[mergify]

backport 9.2 9.3

✅ Backports have been created

Details

• #48988 [9.2](backport #48432) metricbeats: update monitor mappings to include new TBS metrics has been created for branch 9.2
• #48989 [9.3](backport #48432) metricbeats: update monitor mappings to include new TBS metrics has been created for branch 9.3

[ericywl]

@Mergifyio backport 8.19

[mergify]

backport 8.19

✅ Backports have been created

Details

• #49457 [8.19](backport #48432) metricbeats: update monitor mappings to include new TBS metrics has been created for branch 8.19 but encountered conflicts

Cherry-pick of 112c334 has failed:

On branch mergify/bp/8.19/pr-48432
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 112c3349e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1769485799-tbs-monitoring.yaml
	modified:   metricbeat/module/beat/_meta/fields.yml
	modified:   metricbeat/module/beat/fields.go
	modified:   metricbeat/module/beat/stats/_meta/fields.yml
	modified:   metricbeat/module/beat/stats/_meta/test/apm-server.stats.712.json

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   docs/reference/metricbeat/exported-fields-beat.md

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally