Use beat receiver subcomponent status for status translation

[swiatekm]

What does this PR do?

Uses subcomponent status metadata emitted by beats receivers to set unit and stream status for running components. This brings beat receiver status reporting up to par with beats processes.

As part of the implementation, I moved the otel status event serialization logic to its own package, so I can reuse it in status translation. Subcomponent statuses come in an untyped map and need to be converted to a status.Event struct before we can calculate the statuses.

This PR currently imports a patched status aggregation package from collector contrib, as the upstream PR adding the necessary functionality hasn't been merged yet. This uses our official fork and we'll have to maintain it through otel contrib upgrades until it lands and is released.

Why is it important?

Beats receivers and beats processes should be at feature parity. In particular, individual units and streams should have their status set based on the underlying beats input or module, and not based on the aggregated status of the whole receiver. If a system/metrics module emits a diagnostic message about some targets being unreachable (because they're processes owned by another uses and agent isn't running as root, for example), then this should work as expected too with beats receivers.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

Build agent locally and try the following config. You should see degraded statuses on the streams.

agent:
  logging:
    to_stderr: true
    level: error
  monitoring:
    enabled: false
inputs:
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
    id: unique-system-metrics-input-cpu
  - data_stream:
      dataset: system.process
    metricsets:
    - process
    id: unique-system-metrics-input-process
  type: system/metrics
  use_output: default
- data_stream:
    namespace: default
  id: unique-system-metrics-input-2
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
    id: unique-system-metrics-input-2-cpu
  - data_stream:
      dataset: system.process
    metricsets:
    - process
    id: unique-system-metrics-input-2-process
    degrade_on_partial: true
  type: system/metrics
  use_output: default
outputs:
  default:
    username: elastic
    password: elastic
    hosts:
    - 127.0.0.1:9200
    type: elasticsearch
    status_reporting:
      enabled: false

Related issues

• Relates https://github.com/elastic/ingest-dev/issues/5742

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/beat-receiver-subcomponent-status upstream/feat/beat-receiver-subcomponent-status
git merge upstream/main
git push upstream feat/beat-receiver-subcomponent-status

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/beat-receiver-subcomponent-status upstream/feat/beat-receiver-subcomponent-status
git merge upstream/main
git push upstream feat/beat-receiver-subcomponent-status

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 27a866f

Failed CI Steps

• check-ci

History

• 💚 Build #33462 succeeded 9730e27
• 💔 Build #33452 failed 2b52443
• 💔 Build #33347 failed 190cf65
• 💔 Build #33342 failed 9bceeec
• 💔 Build #33339 failed 0cf8d7d
• 💔 Build #33280 failed 6f0e71a

cc @swiatekm

[cmacknz]

Notice needs to be updated, otherwise LGTM (again - this one is cursed apparently)