Disable gRPC compression #4429
Disable gRPC compression #4429
Conversation
carles-grafana
requested review from
joe-elliott,
mdisibio,
mapno,
yvrhdn,
zalegrala,
electron0zero,
ie-pham,
stoewer and
javiermolinar
as code owners
CHANGELOG.md
Outdated
| @@ -1,5 +1,6 @@ | |||
| ## main / unreleased | |||
| * [FEATURE] tempo-cli: support dropping multiple traces in a single operation [#4266](https://github.com/grafana/tempo/pull/4266) (@ndk) | |||
| * [CHANGE] Disable gRPC compression in the querier and distributor. [#4429] (https://github.com/grafana/tempo/pull/4429) (@carles-grafana) | |||
There was a problem hiding this comment.
Let's add some more details here. Something like:
Disable gRPC compression in the querier and distributor for performance reasons.
If you would like to reenable we recommend snappy. Use the following settings:
querier:
<whatever the path to the setting is>
distributor:
<whatever the path to the setting is>
|
Looks like there's a changelog conflict. Get that cleaned up and we'll merge! |
Our benchmark suggests that without compression, queriers and distributors use less CPU and memory. This setting is configured at the client end. That is, in the querier and in the ingester and metrics_generator clients of the distributor.
carles-grafana
force-pushed
the
disable-grpc-compression
branch
from
395a16d to
f2b9f9f
Compare
| @@ -116,9 +116,9 @@ func (c *Config) RegisterFlagsAndApplyDefaults(prefix string, f *flag.FlagSet) { | |||
|
|
|||
| // Everything else | |||
| flagext.DefaultValues(&c.IngesterClient) | |||
| c.IngesterClient.GRPCClientConfig.GRPCCompression = "snappy" | |||
| c.IngesterClient.GRPCClientConfig.GRPCCompression = "" | |||
There was a problem hiding this comment.
nit: since this is changing the default values, let's update docs/sources/tempo/configuration/manifest.md as well.
There was a problem hiding this comment.
I'm writing a script to generate it automatically and to the check it in CI. I will add it in another PR.
|
some before/after benchmarks would have been nice ! otherwise everyone is aware of the tradeoff. |
In our testing environment we've seen less CPU usage in queries that return a high number of series. Likewise, in our distributors we saw a significant drop after disabling compression. |
knylander-grafana
removed
the
type/docs
Our benchmark suggests that without compression, queriers and distributors use less CPU and memory.
This setting is configured at the client end.
That is, in the querier and in the ingester and metrics_generator clients of the distributor.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]