[8.19] (backport #11186) Allow setting the otel runtime per input type

[mergify]

What does this PR do?

Allows the runtime in which an input runs to be set per input type in a new agent.internal.runtime configuration section. There's three levels at which the runtime can be set, in order of increasing priority:

• The default for all input types.
• The default for all input types running in a specific beat.
• The default for a single input type.

Here's how the configuration might look like with all options set:

agent:
  internal:
    runtime:
      default: otel
      filebeat:
        default: otel
        filestream: process
      metricbeat:
        default: process
        system/metrics: otel

The implementation is almost completely confined to the component package.

Why is it important?

We intend to progressively make the otel runtime the default in future major releases, but we also want to let users switch back to the process runtime if they encounter bugs. These settings are intended to be transient - eventually, they'll be removed,.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

Build agent locally and try playing around with the settings in the following config file:

agent:
  logging:
    to_stderr: true
  internal:
    runtime:
      default: otel
      filebeat.filestream: process
inputs:
- id: filestream-filebeat
  type: filestream
  paths:
  - /tmp/nonexistent1
  prospector.scanner.fingerprint.enabled: false
  file_identity.native: ~
  use_output: default
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
  type: system/metrics
  use_output: default
outputs:
  default:
    username: elastic
    password:
    hosts:
    - 127.0.0.1:9200
    type: elasticsearch

Related issues

• Relates https://github.com/elastic/ingest-dev/issues/5567

---

This is an automatic backport of pull request #11186 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of f5ed777 has failed:

On branch mergify/bp/8.19/pr-11186
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit f5ed7775f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   _meta/config/common.reference.p2.yml.tmpl
	new file:   changelog/fragments/1764270813-runtime-per-input-type.yaml
	modified:   elastic-agent.reference.yml
	modified:   internal/pkg/agent/application/coordinator/coordinator_test.go
	modified:   internal/pkg/agent/application/coordinator/coordinator_unit_test.go
	modified:   internal/pkg/agent/application/coordinator/diagnostics_test.go
	new file:   internal/pkg/agent/configuration/internal.go
	modified:   internal/pkg/agent/configuration/settings.go
	modified:   internal/pkg/otel/translate/otelconfig.go
	modified:   pkg/component/load.go
	modified:   pkg/component/load_test.go
	modified:   testing/integration/ess/beat_receivers_test.go
	modified:   testing/integration/ess/diagnostics_test.go
	modified:   testing/integration/ess/otel_test.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   internal/pkg/agent/application/coordinator/coordinator.go
	both modified:   internal/pkg/agent/cmd/inspect.go
	deleted by us:   internal/pkg/agent/install/componentvalidation/validation.go
	both modified:   internal/pkg/agent/install/uninstall.go
	both modified:   pkg/component/component.go
	both modified:   pkg/component/component_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.19/pr-11186 upstream/mergify/bp/8.19/pr-11186
git merge upstream/8.19
git push upstream mergify/bp/8.19/pr-11186

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 23efabf

Failed CI Steps

• Win2022:sudo:default
• Win2022:sudo:default
• arm:sudo: default
• x86_64:sudo: default - platform-ingest-elastic-agent-debian-11-1762801856

History

• 💔 Build #31270 failed 3fa7168
• 💔 Build #31254 failed ec9e1ad

cc @swiatekm