Skip to content

Conversation

@chaudum
Copy link
Contributor

@chaudum chaudum commented Nov 5, 2024

What this PR does / why we need it:

A lot of CPU time is spent in decoding the meta files, which are JSON files.

See this CPU profile from a trace:

screenshot_20241105_101340

github.com/json-iterator/go is a drop-in replacement for the Go stdlib encoding/json.

Benchmark

$ benchstat old.txt new.txt 
goos: linux
goarch: amd64
pkg: github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
                                     │   old.txt    │               new.txt                │
                                     │    sec/op    │    sec/op     vs base                │
_Fetcher_processMetasCacheResponse-8   4.318µ ± 12%   2.095µ ± 49%  -51.49% (p=0.000 n=10)

                                     │   old.txt   │              new.txt               │
                                     │    B/op     │    B/op     vs base                │
_Fetcher_processMetasCacheResponse-8   1301.0 ± 0%   869.0 ± 0%  -33.21% (p=0.000 n=10)

                                     │  old.txt   │              new.txt               │
                                     │ allocs/op  │ allocs/op   vs base                │
_Fetcher_processMetasCacheResponse-8   27.00 ± 0%   15.00 ± 0%  -44.44% (p=0.000 n=10)

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR
@chaudum chaudum requested a review from a team as a code owner November 5, 2024 09:16
@chaudum chaudum force-pushed the chaudum/replace-json-lib-for-metas branch from 9ef2c8e to d0ba985 Compare November 5, 2024 09:20
@pull-request-size pull-request-size bot added size/M and removed size/XS labels Nov 5, 2024
Copy link
Contributor

@salvacorts salvacorts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Happy to merge once the CI is happy.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@chaudum chaudum force-pushed the chaudum/replace-json-lib-for-metas branch from 5a45085 to 7bb4b8b Compare November 5, 2024 10:52
@chaudum chaudum merged commit 57f16a0 into main Nov 5, 2024
59 checks passed
@chaudum chaudum deleted the chaudum/replace-json-lib-for-metas branch November 5, 2024 11:03
chaudum added a commit that referenced this pull request Nov 6, 2024
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this pull request Nov 6, 2024
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this pull request Nov 7, 2024
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 participants