NGINX OpenTelemetry Input Package

[ishleenk17]

This is a draft version of NGINX OpenTelemetry Input Package.
Since this is the very first OTEL Input package, this PR is being used to figure out the structure, template, what works what doesn't while creating packages.

FYI, using HTTPChecker example as reference

Relates: https://github.com/elastic/obs-integration-team/issues/622

[ishleenk17]

@jsoriano @cmacknz : Could you please review this first OTEL input package

Point to discuss:

1. We need to have a check in data_stream.dataset. It should always be same as the receiver name (for metrics) for the content package to autoinstall. For logs we need to see how will we ensure about the datastream field.
2. Under compatibility section of documentation, what all should we mention or shall we just point to the upstream receiver ? I think pointing to upstream receiver might not be right as our EDOT collector might not be at that the upstream collector label ?
3. Ideally we don't need base-fileds.yml. But elastic package doesn't allow to build the package without that. We should remove this check for these fields in OTEL Input package.
4. If we are updating this, we should also update the corresponding content package documentation. We need to discuss how. Currently it contains how a collecotr config looks like. But since now that info is there in the input package. Should we have that in content package ? We might need it if someone tries to install that content package via upstream collector and not EDOT. Thoughts ?

cc: @tommyers-elastic

[jsoriano]

This file can be removed. OTel input packages rely on pre-built component templates.

If this is the only fields file, build will fail, till we release elastic/package-spec#994 (probably this week).

[ishleenk17]

Yes, this is mostly going to be the only file under fields. I don't expect any mappings to go in here.
Do we actually expect predefined mappings for OTEL Input packages.
Or we can rely on the otel template and the dynamic mappings ?

[jsoriano]

Do we actually expect predefined mappings for OTEL Input packages.
Or we can rely on the otel template and the dynamic mappings ?

We can rely on the pre-installed template and the dynamic mappings it includes.

We still have the option of defining fields if needed in some case, but defining fields will be completely optional for OTel input packages. It will be actually expected for OTel input packages to don't contain any field definition.

[jsoriano]

Did you consider removing this file? You can leave it empty or wait for next elastic-package release (coming soon) if you have problems with validation.

[ishleenk17]

@jsoriano On removing the base-fields.yml

1. file "/Users/ishleenkaur/go/ish_12Oct_integ/ish_integrations/build/packages/nginx_otel_input-0.0.1.zip/fields/base-fields.yml" is invalid: file is empty, but media type is defined

2. On having this file empty, we get below message:

Error: building package failed: invalid content found in built zip package: found 4 validation errors:

1. expected field "data_stream.dataset" with type "constant_keyword" not found
2. expected field "data_stream.namespace" with type "constant_keyword" not found
3. expected field "@timestamp" with type "date" not found
4. expected field "data_stream.type" with type "constant_keyword" not found

So for now we need to have this file with the fields, or we wait for the next release of elastic package.

[fred-maussion]

This is working as expected with the latest elastic-package version where the build is successful without the base-fields.yml and the fields folder
elastic-package v0.117.1 version-hash undefined (build time: unknown)

[ishleenk17]

@jsoriano : COuld you pelase rereview this ?

[jsoriano]

Looks good overall, but there are a couple of threads that seem to lack a resolution.

[jsoriano]

Did you consider removing this file? You can leave it empty or wait for next elastic-package release (coming soon) if you have problems with validation.

[ishleenk17]

After discussing with @jsoriano , we will proceed with the merge after the new version of elastic package is released. It will help us removing the bas-fields.yml file.

[ishleenk17]

After discussing with @jsoriano , we will proceed with the merge after the new version of elastic package is released. It will help us removing the bas-fields.yml file.

With new elastic package, packages pass CI even without any fields related files.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 8fce6c6

History

• 💚 Build #34710 succeeded 007e22d
• 💚 Build #34705 succeeded 4999153
• 💚 Build #34017 succeeded 40cbd02
• 💚 Build #33781 succeeded 581b15a
• 💔 Build #33780 failed 45c7269
• 💚 Build #33242 succeeded 87a3579

cc @ishleenk17

[tommyers-elastic]

thanks ishleen. approving, but i think we should add the component identifiers before we merge.

[lalit-satapathy]

Triaging approval.

[elastic-vault-github-plugin-prod]

Package nginx_otel_input - 0.1.0 containing this change is available at https://epr.elastic.co/package/nginx_otel_input/0.1.0/