[otel/kube-stack]: Add gateway collector by rogercoll · Pull Request #6444 · elastic/elastic-agent

rogercoll · 2024-12-27T13:36:53Z

What does this PR do?

This PR adds a new K8s deployment of the EDOT collector named "gateway". The main purpose of this new deployment is to simplify the daemonset collector configuration and unify managed/self-managed scenarios. The gateway collector configuration contains all Elastic's custom Otel components needed for the signals transformations in self-managed scenarios, which are currently configured in the daemonset collector.

Elastic configured components in the "Gateway" collectors (previously in the daemonset):

signaltometrics
elasticinframetrics
elastictrace
lsminterval

Another important change is that the Gateway collectors will forward the Otel data to Elasticsearch, the daemonset and cluster collectors configurations have been updated to export all collected data to the corresponding Gateway OTLP endpoint. Although the daemonset collectors are still configured to collect the auto-instrumentation OTLP data, the data is load balanced (loadbalancing exporter) to the gateway collectors based on the service name.

Additional context: https://github.com/elastic/opentelemetry-dev/issues/587

Why is it important?

The key benefit of this architecture is to decouple data collection from data transformations (e.g. APM enrichment and aggregations), for managed scenarios, users would just need to remove (or comment) the "gateway" collector configuration. Note that moving the data processing from a k8s "daemonset" to a "deployment", it eases its horizontal scaling.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2024-12-27T13:37:32Z

This pull request does not have a backport label. Could you fix it @rogercoll? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-12-27T13:37:32Z

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

michalpristas · 2024-12-30T07:32:19Z

please link an issue if such exists

deploy/helm/edot-collector/kube-stack/values.yaml

rogercoll · 2025-01-08T11:25:21Z

This new configuration requires the loadbalancing and routing components that were added in #6210 and #6315

elasticmachine · 2025-01-08T11:25:50Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

felixbarny

LGTM! Let's get it in and iterate from there.

deploy/helm/edot-collector/kube-stack/values.yaml

rogercoll · 2025-01-24T10:35:13Z

@elastic/elastic-agent-control-plane PR back to ready for review

elastic-sonarqube · 2025-01-24T15:35:00Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

* feat: move telemetry aggregation and forwarding to gateway * ci: use Elastic envs in gateway * chore: add changelog entry * fix: format values file * feat: add apm loadbalancing * chore: increase resource limits * revert resource limits increase * chore: remove config warnings * docs: add Gateway collectors section * revert: enable daemonset storagechecks * rename metrics/otel pipeline and use signaltometrics * unify k8s and host metrics pipelines * use default traceID as loadbalancing routing_key * chore: reuse k8s integration test helpers * format values with Helm linter * replace loadbalancing in favor of headless otlp * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * rename k8s values options helper function * move process attributes remove processor to gateway * add batch processor for aggregation pipeline * enable compression for cluster otlp connections * chore: remove elastic endpoint references * fix: do not generate service's signals for non apm data * Revert "fix: do not generate service's signals for non apm data" This reverts commit ffa6620. * fix: set agent.name as edot-collector * fix: enable daemon hostNetwork * set unknown as default signaltometrics agent.name resource attribute * remove signaltometrics for metrics-only services --------- Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> (cherry picked from commit daed81e)

* feat: move telemetry aggregation and forwarding to gateway * ci: use Elastic envs in gateway * chore: add changelog entry * fix: format values file * feat: add apm loadbalancing * chore: increase resource limits * revert resource limits increase * chore: remove config warnings * docs: add Gateway collectors section * revert: enable daemonset storagechecks * rename metrics/otel pipeline and use signaltometrics * unify k8s and host metrics pipelines * use default traceID as loadbalancing routing_key * chore: reuse k8s integration test helpers * format values with Helm linter * replace loadbalancing in favor of headless otlp * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * rename k8s values options helper function * move process attributes remove processor to gateway * add batch processor for aggregation pipeline * enable compression for cluster otlp connections * chore: remove elastic endpoint references * fix: do not generate service's signals for non apm data * Revert "fix: do not generate service's signals for non apm data" This reverts commit ffa6620. * fix: set agent.name as edot-collector * fix: enable daemon hostNetwork * set unknown as default signaltometrics agent.name resource attribute * remove signaltometrics for metrics-only services --------- Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> (cherry picked from commit daed81e) Co-authored-by: Roger Coll <rogercoll@protonmail.com>

* feat: move telemetry aggregation and forwarding to gateway * ci: use Elastic envs in gateway * chore: add changelog entry * fix: format values file * feat: add apm loadbalancing * chore: increase resource limits * revert resource limits increase * chore: remove config warnings * docs: add Gateway collectors section * revert: enable daemonset storagechecks * rename metrics/otel pipeline and use signaltometrics * unify k8s and host metrics pipelines * use default traceID as loadbalancing routing_key * chore: reuse k8s integration test helpers * format values with Helm linter * replace loadbalancing in favor of headless otlp * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * rename k8s values options helper function * move process attributes remove processor to gateway * add batch processor for aggregation pipeline * enable compression for cluster otlp connections * chore: remove elastic endpoint references * fix: do not generate service's signals for non apm data * Revert "fix: do not generate service's signals for non apm data" This reverts commit ffa6620. * fix: set agent.name as edot-collector * fix: enable daemon hostNetwork * set unknown as default signaltometrics agent.name resource attribute * remove signaltometrics for metrics-only services --------- Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> (cherry picked from commit daed81e)

* feat: move telemetry aggregation and forwarding to gateway * ci: use Elastic envs in gateway * chore: add changelog entry * fix: format values file * feat: add apm loadbalancing * chore: increase resource limits * revert resource limits increase * chore: remove config warnings * docs: add Gateway collectors section * revert: enable daemonset storagechecks * rename metrics/otel pipeline and use signaltometrics * unify k8s and host metrics pipelines * use default traceID as loadbalancing routing_key * chore: reuse k8s integration test helpers * format values with Helm linter * replace loadbalancing in favor of headless otlp * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * Update testing/integration/otel_helm_test.go Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> * rename k8s values options helper function * move process attributes remove processor to gateway * add batch processor for aggregation pipeline * enable compression for cluster otlp connections * chore: remove elastic endpoint references * fix: do not generate service's signals for non apm data * Revert "fix: do not generate service's signals for non apm data" This reverts commit ffa6620. * fix: set agent.name as edot-collector * fix: enable daemon hostNetwork * set unknown as default signaltometrics agent.name resource attribute * remove signaltometrics for metrics-only services --------- Co-authored-by: Panos Koutsovasilis <koutsobill@hotmail.com> (cherry picked from commit daed81e) Co-authored-by: Roger Coll <rogercoll@protonmail.com>

rogercoll added 3 commits December 27, 2024 14:25

feat: move telemetry aggregation and forwarding to gateway

67371a7

ci: use Elastic envs in gateway

9d84b4a

chore: add changelog entry

662557f

mergify bot assigned rogercoll Dec 27, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Dec 27, 2024

fix: format values file

36aeb8e

michalpristas added the enhancement New feature or request label Dec 30, 2024

swiatekm added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Dec 31, 2024

rogercoll added 9 commits January 3, 2025 15:12

feat: add apm loadbalancing

ca4b80f

chore: increase resource limits

f990086

Merge branch 'add_gateway_collector_lb' into add_gateway_collector

f0a8c5c

revert resource limits increase

7d50d2a

Merge branch 'main' into add_gateway_collector

f8a8581

chore: remove config warnings

5053e0b

Merge branch 'main' into add_gateway_collector

16247d6

docs: add Gateway collectors section

238d6c3

revert: enable daemonset storagechecks

b0fd91d

rogercoll commented Jan 8, 2025

View reviewed changes

deploy/helm/edot-collector/kube-stack/values.yaml Show resolved Hide resolved

Merge branch 'main' into add_gateway_collector

e633c52

rogercoll marked this pull request as ready for review January 8, 2025 11:25

rogercoll requested a review from a team as a code owner January 8, 2025 11:25

rogercoll requested review from kaanyalti and pkoutsovasilis January 8, 2025 11:25

rogercoll requested review from gizas and tetianakravchenko January 8, 2025 11:26

rogercoll added 4 commits January 23, 2025 12:41

fix: enable daemon hostNetwork

6179c9e

Merge branch 'main' into add_gateway_collector

c98d3c6

set unknown as default signaltometrics agent.name resource attribute

7cdb994

Merge branch 'main' into add_gateway_collector

9825bad

felixbarny approved these changes Jan 24, 2025

View reviewed changes

deploy/helm/edot-collector/kube-stack/values.yaml Outdated Show resolved Hide resolved

remove signaltometrics for metrics-only services

c83cd8f

Merge branch 'main' into add_gateway_collector

47356d8

rogercoll mentioned this pull request Jan 24, 2025

chore: bump kube-stack Helm chart onboarding version elastic/kibana#208217

Merged

9 tasks

rogercoll requested review from pkoutsovasilis and swiatekm January 24, 2025 16:12

pkoutsovasilis approved these changes Jan 27, 2025

View reviewed changes

swiatekm approved these changes Jan 27, 2025

View reviewed changes

rogercoll merged commit daed81e into elastic:main Jan 27, 2025
14 checks passed

mergify bot mentioned this pull request Jan 27, 2025

[8.x](backport #6444) [otel/kube-stack]: Add gateway collector #6606

Merged

8 tasks

felixbarny mentioned this pull request Feb 19, 2025

Introduce elasticapmconnector elastic/opentelemetry-collector-components#367

Merged

rogercoll added the backport-8.17 Automated backport with mergify label Feb 24, 2025

mergify bot mentioned this pull request Feb 24, 2025

[8.17](backport #6444) [otel/kube-stack]: Add gateway collector #6990

Merged

8 tasks

This was referenced Mar 13, 2025

Fix exporter configuration in kube-stack values #7380

Merged

Add functional tests for the opentelemetry-kube-stack configuration #7381

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[otel/kube-stack]: Add gateway collector#6444

[otel/kube-stack]: Add gateway collector#6444
rogercoll merged 40 commits intoelastic:mainfrom
rogercoll:add_gateway_collector

rogercoll commented Dec 27, 2024 •

edited

Loading

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

michalpristas commented Dec 30, 2024

Uh oh!

rogercoll commented Jan 8, 2025

elasticmachine commented Jan 8, 2025

felixbarny left a comment

Uh oh!

rogercoll commented Jan 24, 2025

elastic-sonarqube bot commented Jan 24, 2025

Uh oh!

Labels

9 participants

Conversation

rogercoll commented Dec 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

michalpristas commented Dec 30, 2024

Uh oh!

rogercoll commented Jan 8, 2025

elasticmachine commented Jan 8, 2025

felixbarny left a comment

Choose a reason for hiding this comment

Uh oh!

rogercoll commented Jan 24, 2025

elastic-sonarqube bot commented Jan 24, 2025

Quality Gate passed

Uh oh!

Labels

9 participants

rogercoll commented Dec 27, 2024 •

edited

Loading