Complete OTel configuration pipelines with an exporter#233090
Complete OTel configuration pipelines with an exporter#233090jsoriano merged 32 commits intoelastic:mainfrom
Conversation
| return attachExporter(config, dataOutput); | ||
| } | ||
|
|
||
| function addSuffixToOtelcolComponentsConfig(type: string, components: {[key:string]:any}, suffix: string) { |
There was a problem hiding this comment.
TypeScript has a Record<string, any> utility type for objects like this
|
Opening for review, from here we need to update fleet-server to inject credentials for the outputs. |
botelastic
bot
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
|
Fleet Server changes in elastic/fleet-server#5469. Moving back to draft as we also need to add the dataset fields for routing, as described in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/README.md#elasticsearch-document-routing. |
jsoriano
added
backport:skip
jsoriano
force-pushed
the
otel-exporters
branch
from
c2c8458 to
63f3ccc
Compare
|
Open again for review. |
x-pack/platform/plugins/shared/fleet/server/services/agent_policies/full_agent_policy.ts
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/fleet/server/services/agent_policies/otel_collector.ts
Outdated
Show resolved
Hide resolved
…icies/otel_collector.ts Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
|
@elasticmachine merge upstream |
💔 Build Failed
Failed CI StepsTest Failures
Metrics [docs]
History
cc @jsoriano |
| context = 'span'; | ||
| break; | ||
| default: | ||
| throw new Error(`unexpected data stream type ${type}`); |
There was a problem hiding this comment.
nit: maybe you can use FleetError here
| let patchedYaml = yamlFromCompiledTemplate; | ||
| if ( | ||
| experimentalFeature.enableOtelIntegrations && | ||
| inputType === OTEL_COLLECTOR_INPUT_TYPE && |
There was a problem hiding this comment.
Curious why this check was removed? Was it moved somewhere else?
There was a problem hiding this comment.
I moved all the logic for processing inputs to generateOtelcolConfig, that is now on its own file.
In part it was because while supporting policies with multiple packages I found that the policy id was not enough as identifier, as there could be several streams of the same kind in the same policy. So the suffix used here later needed to be appended to other ids in generateOtelcolConfig. I found it easier to have everything in the same place, and specially easier to test.
Now generateOtelcolConfig receives the unmodified inputs and takes care of generating the unique ids, merging the configs and attaching the connectors and exporters. We don't need to propagate the ids so much, or modify the compilation step, and everything is encapsulated in the same place.
I wasn't sure about moving this out of the compilation step, not sure if we will be missing something later by doing this.
Regarding the feature flag, in principle it is still checked in https://github.com/elastic/kibana/pull/233090/files#diff-f23ad8d1dbe97e73058031bebb5482e9c4272780c63d5d601a60c867d208a99bL164.
| const experimentalFeature = appContextService.getExperimentalFeatures(); | ||
|
|
||
| return inputs.map((input) => { | ||
| if (experimentalFeature.enableOtelIntegrations && input.type === OTEL_COLLECTOR_INPUT_TYPE) { |
There was a problem hiding this comment.
Same thing, as I was moving all the id generation to generateOtelcolConfig I removed all the code that was using the otelcolSuffixId in the compile steps.
Though looking here maybe I removed too much, should we ensure that compiled_input is empty for otelcol inputs?
There was a problem hiding this comment.
I checked with the httpcheck integration and compiled_input is still empty, it shouldn't be an issue regardless.
|
Tested the code locally, the config seems good to me: 
|
- Merge configuration of multiple inputs in a single configuration. - Add a forward connector as exporter for each input pipeline. - Generate an exporter configuration based on the assigned data output. - Add a pipeline with all the forward connectors as receivers and the generated exporter as exporter. - Add a transform processor to each pipeline with the attributes for dynamic routing.
- Merge configuration of multiple inputs in a single configuration. - Add a forward connector as exporter for each input pipeline. - Generate an exporter configuration based on the assigned data output. - Add a pipeline with all the forward connectors as receivers and the generated exporter as exporter. - Add a transform processor to each pipeline with the attributes for dynamic routing.
- Merge configuration of multiple inputs in a single configuration. - Add a forward connector as exporter for each input pipeline. - Generate an exporter configuration based on the assigned data output. - Add a pipeline with all the forward connectors as receivers and the generated exporter as exporter. - Add a transform processor to each pipeline with the attributes for dynamic routing.
6 participants
Summary
Complete composition of OTel configuration pipelines:
Closes https://github.com/elastic/ingest-dev/issues/5712.
How to test?
Follow instructions in elastic/fleet-server#5469, this branch is also needed.
Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
release_note:breakinglabel should be applied in these situations.release_note:*label is applied per the guidelinesbackport:*labels.Identify risks
Low risk, changes are for an unreleased feature behind a feature flag.