Skip to content

fix: data race in chunk client hedging tests #15466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 18, 2024
Merged

fix: data race in chunk client hedging tests #15466

merged 1 commit into from
Dec 18, 2024

Conversation

@na--
Copy link
Contributor

@na-- na-- commented Dec 18, 2024

What this PR does / why we need it:

Before these changes, the tests here failed with a bunch of data races, e.g.:

WARNING: DATA RACE
Read at 0x000000eb74e0 by goroutine 17:
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.(*winnerTrackingRoundTripper).RoundTrip()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging.go:160 +0xa4
  github.com/cristalhq/hedgedhttp.(*hedgedTransport).RoundTrip.func2()
      github.com/cristalhq/hedgedhttp@v0.9.1/hedged.go:211 +0xa4
  github.com/cristalhq/hedgedhttp.runInPool.func1()
      github.com/cristalhq/hedgedhttp@v0.9.1/hedged.go:305 +0x39

Previous write at 0x000000eb74e0 by goroutine 22:
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.initMetrics()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging.go:39 +0x2cf
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.resetMetrics()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging_test.go:25 +0x1f5
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.TestHedgingRateLimit()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging_test.go:64 +0x33
...

WARNING: DATA RACE
Read at 0x00c00023c750 by goroutine 17:
  github.com/prometheus/client_golang/prometheus.(*MetricVec).GetMetricWithLabelValues()
      github.com/prometheus/client_golang@v1.20.5/prometheus/vec.go:213 +0x55
  github.com/prometheus/client_golang/prometheus.(*CounterVec).GetMetricWithLabelValues()
      github.com/prometheus/client_golang@v1.20.5/prometheus/counter.go:249 +0x57
  github.com/prometheus/client_golang/prometheus.(*CounterVec).WithLabelValues()
      github.com/prometheus/client_golang@v1.20.5/prometheus/counter.go:282 +0xfb
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.(*winnerTrackingRoundTripper).RoundTrip()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging.go:160 +0x99
  github.com/cristalhq/hedgedhttp.(*hedgedTransport).RoundTrip.func2()
      github.com/cristalhq/hedgedhttp@v0.9.1/hedged.go:211 +0xa4
  github.com/cristalhq/hedgedhttp.runInPool.func1()
      github.com/cristalhq/hedgedhttp@v0.9.1/hedged.go:305 +0x39

Previous write at 0x00c00023c750 by goroutine 22:
  github.com/prometheus/client_golang/prometheus.NewMetricVec()
      github.com/prometheus/client_golang@v1.20.5/prometheus/vec.go:48 +0x175
  github.com/prometheus/client_golang/prometheus.v2.NewCounterVec()
      github.com/prometheus/client_golang@v1.20.5/prometheus/counter.go:213 +0x9d
  github.com/prometheus/client_golang/prometheus.NewCounterVec()
      github.com/prometheus/client_golang@v1.20.5/prometheus/counter.go:195 +0x2be
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.initMetrics()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging.go:39 +0x192
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.resetMetrics()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging_test.go:25 +0x1f5
  github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging.TestHedgingRateLimit()
      github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging/hedging_test.go:64 +0x33
...
@na-- na-- requested a review from a team as a code owner December 18, 2024 12:55
@na-- na-- force-pushed the ned/fix-data-race-05 branch from 638c5b9 to d2b856e Compare December 18, 2024 14:20
slim-bean
slim-bean reviewed Dec 18, 2024
View reviewed changes
slim-bean
slim-bean requested changes Dec 18, 2024
View reviewed changes
Copy link
Collaborator

@slim-bean slim-bean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question if we can simplify this a bit? see comment
slim-bean
slim-bean approved these changes Dec 18, 2024
View reviewed changes
Copy link
Collaborator

@slim-bean slim-bean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ship it
@na-- na-- merged commit fad06ee into main Dec 18, 2024
58 checks passed
@na-- na-- deleted the ned/fix-data-race-05 branch December 18, 2024 21:17
This was referenced Dec 23, 2024
Closed
Closed
@loki-gh-app loki-gh-app bot mentioned this pull request Jan 6, 2025
Closed
mveitas pushed a commit to mveitas/loki that referenced this pull request Jan 6, 2025
@na-- @matt-veitas-eb
fix: data race in chunk client hedging tests (grafana#15466)
2e5972a
@loki-gh-app loki-gh-app bot mentioned this pull request Jan 13, 2025
Closed
@loki-gh-app loki-gh-app bot mentioned this pull request Jan 20, 2025
Closed
This was referenced Feb 3, 2025
Closed
Merged
Closed
@loki-gh-app loki-gh-app bot mentioned this pull request Feb 12, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/S

2 participants

@na-- @slim-bean