EDOT collector: include the forward connector. by cmacknz · Pull Request #8753 · elastic/elastic-agent

cmacknz · 2025-06-30T21:06:18Z

Relates [Fleet] Support input packages with OTel configuration kibana#224472
Relates New httpcheck package based on the Otel collector httpcheck receiver integrations#14315

https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector

Include the forwardconnector, which we are highly likely to need for OTel native integrations to allow for unique processors per receiver without having to have unique pipelines with duplicate exporters.

To test this PR I used the following configuration showing how to have 2 filelog receivers that could have separate processors applied but export via the same exporter.

receivers:
  filelog:
    include: [ /var/log/system.log ]
    start_at: beginning
  filelog/2:
    include: [ /var/log/wifi.log ]
    start_at: beginning

processors:
  resource:
    attributes:
    - key: service.name
      action: insert
      value: elastic-otel-test

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 10000
    sampling_thereafter: 10000

connectors:
  forward:

extensions:
  memory_limiter:
    check_interval: 1s
    limit_mib: 700
    spike_limit_mib: 180
  health_check:
    endpoint: "localhost:13133"
    path: "/health/status"

service:
  extensions: [health_check, memory_limiter]
  pipelines:
    logs/system:
      receivers: [filelog]
      processors: [resource]
      exporters: [forward]
    logs/wifi:
      receivers: [filelog/2]
      processors: [resource]
      exporters: [forward]
    logs/export:
      receivers: [forward]
      exporters: [debug]

https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector

elasticmachine · 2025-06-30T21:06:21Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

pkoutsovasilis

Indeed forwardconnector seems that it does the trick, so LGTM. Only one minor thingie @cmacknz please run mage fmt and repush 😄

pchila

Code looks ok, I wonder if the new component should be listed in internal/pkg/otel/core-components.yaml as introduced by PR #8530

Holding off approval till CI is green

cmacknz · 2025-07-02T14:29:55Z

Core components means users can open SDHs against them and we'll try to fix them. I don't think we we are at the point where we need that for the foward connector as we don't depend on it for anything yet.

swiatekm · 2025-07-02T16:28:41Z

@cmacknz could you add this component to this file? That way we can be confident it's actually possible to start a collector with it included in a pipeline.

jsoriano

Thanks!

cmacknz · 2025-07-02T18:45:00Z

@cmacknz could you add this component to this file? That way we can be confident it's actually possible to start a collector with it included in a pipeline.

Done in 1d856a2

elastic-sonarqube · 2025-07-02T20:21:40Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-07-02T20:35:36Z

💚 Build Succeeded

Buildkite Build
Commit: 1d856a2

History

💔 Build #23327 failed f6f059e

cc @cmacknz

pchila

LGTM

* EDOT collector: include the forward connector. https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector * Add changelog. * mage fmt. * Add forward connector to all-components.yml (cherry picked from commit 393e2ed)

* EDOT collector: include the forward connector. https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector * Add changelog. * mage fmt. * Add forward connector to all-components.yml (cherry picked from commit 393e2ed) Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

…ng-oblt-cli * upstream: (37 commits) [AutoOps] Add `autoops-es.yml` to Packages (elastic#8728) EDOT collector: include the forward connector. (elastic#8753) Revert "ci: pin elastic-agent version (elastic#8736)" (elastic#8754) bk: retry Start ESS stack for integration tests (elastic#8553) Re-enable TestStandaloneUpgradeRollbackOnRestarts on windows (elastic#8718) removed reviewers from dependabot.yml (elastic#8709) Pass `--header` enrollment option to fleet-server (elastic#8071) Add ability for local output configuration to add to policy configuration (elastic#8766) Bump up github.com/go-viper/mapstructure/v2 dependency (elastic#8764) [Synthetics] Upgrade node to latest lts v20 (elastic#8712) [CI] BK Vault plugin for EC access (elastic#8377) feat: singleTest mage target for each integration test package (elastic#8691) ci: always include 8.19 LTS release branch in snapshots of test versions (elastic#8761) build(deps): bump github.com/elastic/mito from 1.19.0 to 1.20.0 (elastic#8755) chore: fix elastic-agent helm chart examples (elastic#8765) feat: support onboarding-id for kubernetes (elastic#8692) [main][Automation] Bump VM Image version to 1751072471 (elastic#8734) ci: revert deployment_csp_configuration.yaml to create_deployment_csp_configuration.yaml (elastic#8746) kustomize: pin kube-state-metrics version (elastic#8739) ci: pin elastic-agent version (elastic#8736) ...

* upstream: (39 commits) Fix otel extension status reporting (#8696) Refactor user change on service (#8347) [AutoOps] Add `autoops-es.yml` to Packages (#8728) EDOT collector: include the forward connector. (#8753) Revert "ci: pin elastic-agent version (#8736)" (#8754) bk: retry Start ESS stack for integration tests (#8553) Re-enable TestStandaloneUpgradeRollbackOnRestarts on windows (#8718) removed reviewers from dependabot.yml (#8709) Pass `--header` enrollment option to fleet-server (#8071) Add ability for local output configuration to add to policy configuration (#8766) Bump up github.com/go-viper/mapstructure/v2 dependency (#8764) [Synthetics] Upgrade node to latest lts v20 (#8712) [CI] BK Vault plugin for EC access (#8377) feat: singleTest mage target for each integration test package (#8691) ci: always include 8.19 LTS release branch in snapshots of test versions (#8761) build(deps): bump github.com/elastic/mito from 1.19.0 to 1.20.0 (#8755) chore: fix elastic-agent helm chart examples (#8765) feat: support onboarding-id for kubernetes (#8692) [main][Automation] Bump VM Image version to 1751072471 (#8734) ci: revert deployment_csp_configuration.yaml to create_deployment_csp_configuration.yaml (#8746) ...

EDOT collector: include the forward connector.

88dccea

https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector

cmacknz self-assigned this Jun 30, 2025

cmacknz added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Jun 30, 2025

cmacknz requested a review from a team as a code owner June 30, 2025 21:06

cmacknz added the backport-8.19 Automated backport to the 8.19 branch label Jun 30, 2025

cmacknz requested a review from ycombinator June 30, 2025 21:06

cmacknz added the backport-9.1 Automated backport to the 9.1 branch label Jun 30, 2025

cmacknz requested a review from pchila June 30, 2025 21:06

Add changelog.

f6f059e

cmacknz mentioned this pull request Jun 30, 2025

[Fleet] Support input packages with OTel configuration elastic/kibana#224472

Closed

cmacknz requested review from pkoutsovasilis and swiatekm June 30, 2025 21:10

pkoutsovasilis reviewed Jul 1, 2025

View reviewed changes

pchila reviewed Jul 1, 2025

View reviewed changes

mage fmt.

14faa41

jsoriano previously approved these changes Jul 2, 2025

View reviewed changes

Add forward connector to all-components.yml

1d856a2

cmacknz dismissed jsoriano’s stale review via 1d856a2 July 2, 2025 18:44

pchila approved these changes Jul 3, 2025

View reviewed changes

cmacknz merged commit 393e2ed into elastic:main Jul 3, 2025
19 checks passed

mergify bot mentioned this pull request Jul 3, 2025

[8.19] (backport #8753) EDOT collector: include the forward connector. #8833

Merged

mergify bot mentioned this pull request Jul 3, 2025

[9.1] (backport #8753) EDOT collector: include the forward connector. #8834

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EDOT collector: include the forward connector.#8753

EDOT collector: include the forward connector.#8753
cmacknz merged 4 commits intoelastic:mainfrom
cmacknz:cmacknz/edot-add-forward-connector

cmacknz commented Jun 30, 2025 •

edited

Loading

elasticmachine commented Jun 30, 2025

pkoutsovasilis left a comment

pchila left a comment

cmacknz commented Jul 2, 2025

swiatekm commented Jul 2, 2025

jsoriano left a comment

cmacknz commented Jul 2, 2025

elastic-sonarqube bot commented Jul 2, 2025

elasticmachine commented Jul 2, 2025

pchila left a comment

Uh oh!

Labels

6 participants

Conversation

cmacknz commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticmachine commented Jun 30, 2025

pkoutsovasilis left a comment

Choose a reason for hiding this comment

pchila left a comment

Choose a reason for hiding this comment

cmacknz commented Jul 2, 2025

swiatekm commented Jul 2, 2025

jsoriano left a comment

Choose a reason for hiding this comment

cmacknz commented Jul 2, 2025

elastic-sonarqube bot commented Jul 2, 2025

Quality Gate passed

elasticmachine commented Jul 2, 2025

💚 Build Succeeded

History

pchila left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

6 participants

cmacknz commented Jun 30, 2025 •

edited

Loading