Tempo currently has a whole bunch of OTel dependencies at v0.102.1, when I upgrade these to a recent version (v0.116.0) the receiver is not able to receive data anymore in certain configurations.

Background

The cause is a change in behaviour: the OpenTelemetry Collector used to listen on 0.0.0.0 by default. Meaning, if you enabled the receivers with default config, the otlp receiver would for instance listen on 0.0.0.0:4317.

This was brought up as a security risk: open-telemetry/opentelemetry-collector#8510.
Documentation: open-telemetry/opentelemetry-collector//docs/security-best-practices.md#safeguards-against-denial-of-service-attacks

In v0.94.0 a feature gate was added to change this behaviour.
This comment links a lot of PRs related to this work: open-telemetry/opentelemetry-collector#8510 (comment)

Summarized:

• when using an unspecified address the receiver logs a warning
• with the introduction of the feature gate UseLocalHostAsDefaultHost, an unspecified address will be replaced by localhost

In v0.110.0 this flag was made stable.
In v0.112.0 the flag was removed entirely.

How does this impact Tempo?

Most Tempo installations use the receivers with the default config, something like:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

This used to work fine since the receivers defaulted to 0.0.0.0:4317 and 0.0.0.0:4318 respectively. With the changes to replace unspecified address, the receivers now default to localhost:4317 and localhost:4318.
And as a result connections to Tempo running in a Docker container don't work anymore.

To workaround this you need to specify the address you want to bind to explicitly. For instance if Tempo is running in a container with hostname tempo this should work:

# ...
        http:
          endpoint: "tempo:4318"

You can also explicitly bind to 0.0.0.0 still, but this has potential security risks:

# ...
        http:
          endpoint: "0.0.0.0:4318"

If we wish to upgrade the OTel depedency, we need to document this. I don't believe there is a feasible workaround we can do to preserve the original behaviour (nor would it be desirable considering the aforementioned security risks).

To Reproduce

1. Upgrade the OTel dependencies to a version >= 1.102.0 (see for instance Update Jaeger and OTel packages #4406)
2. Rebuild the Docker image
3. Run Tempo with default settings
4. Try to connect to the receiver. If the otlphttp receiver is enabled this normally should work but it doesn't:

$ curl -v -H "Content-type: application/json" -X POST http://localhost:4318/v1/traces -d "{}"

curl: (52) Empty reply from server

If you do the same but run Tempo directly (not containerized) it works:

$ curl -v -H "Content-type: application/json" -X POST http://localhost:4318/v1/traces -d "{}"

{"partialSuccess":{}}%