monitoring: update apm-server metrics collection to avoid conflicts

[isaacaflores2]

Motivation/summary

Currently apm-server.sampling.tail.storage.lsm_size and apm-server.sampling.tail.dynamic_service_groups are never reported together.

Testing locally this is caused because both sets of metrics use the same namespace (metric name prefix) apm-server.sampling but they are created using different instances of a Meter.

The related monitoring func calls addAPMServerMetrics multiple times for each scoped metric. Metric names with the same prefix in different scopes are somehow overwriting each other. This approach opts to collect all "apm-server" metrics and add them to the snapshot once. Another approach would be to update the elastic-agent-lib to prevent metrics from overwriting each other.

Checklist

• View individual metrics documents to confirm reported metrics are correct

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

1. Unit test cover the expected behavior

manual test

1. Run apm-server with TBS enabled
2. Send data to the server ./sendotlp -insecure -endpoint=http://localhost:8200 -secret-token=<token>
3. View metrics via the stats endpoint (http://localhost:5066/stats?pretty) . storage should be visible along with other metrics under apm-server.sampling.tail

"sampling": {
		"tail": {
			"dynamic_service_groups": 0,
			"events": {
				"processed": 3,
				"sampled": 6,
				"stored": 3
			},
			"storage": {
				"lsm_size": 7431,
				"value_log_size": 0
			}
		}
	}

Related issues

Closes #17342
Alternate approach to #17427

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it @isaacaflores2? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b monitoring-stats-apm-server upstream/monitoring-stats-apm-server
git merge upstream/main
git push upstream monitoring-stats-apm-server

[rubvs]

Quickly tested this manually.

• Stats before the PR change

"sampling": {
  "tail": {
    "dynamic_service_groups": 1,
    "events": {
      "processed": 3,
      "stored": 3
    }
  }
}

• Stats after the PR change

"sampling": {
  "tail": {
    "dynamic_service_groups": 1,
    "events": {
      "processed": 3,
      "sampled": 3,
      "stored": 3
    },
    "storage": {
      "lsm_size": 5407,
      "value_log_size": 0
    }
  }
}

• Cat of APM Server config

apm-server:
  host: "127.0.0.1:8200"

  sampling.tail:
    enabled: true
    policies:
      - sample_rate: 1.0

output.elasticsearch:
  hosts: ["https://216c4d53f99e4e48a556e4cfe72cb680.us-central1.gcp.qa.cld.elstc.co:443"]
  username: "elastic"
  password: "<REDACTED>"

http:
  enabled: true
  host: localhost
  port: 5066

[carsonip]

lgtm, thanks for adding the test

[simitt]

Great finding!

[carsonip]

✔️ test-plan-ok

Testing notes

Tested with 9.0.4 BC on ECH. lsm_size and dynamic_service_groups appear in the same monitoring metric document. It does not happen on 9.0.3.