Skip to content

fix: Fix ruler panic related to unset validation scheme - #21400

Merged
paul1r merged 1 commit into
mainfrom
paul1r/fix_ruler_panic
Apr 3, 2026
Merged

fix: Fix ruler panic related to unset validation scheme#21400
paul1r merged 1 commit into
mainfrom
paul1r/fix_ruler_panic

Conversation

@paul1r

@paul1r paul1r commented Apr 3, 2026

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
Without the change in notifier.go, we receive the following stack trace from this test:

--- FAIL: TestNotifierManagerSendWithAlertRelabelReplaceDoesNotPanic (0.00s)
    /Users/progers/dev/src/github.com/grafana/loki/pkg/ruler/base/notifier_test.go:478: 
        	Error Trace:	/Users/progers/dev/src/github.com/grafana/loki/pkg/ruler/base/notifier_test.go:478
        	Error:      	func (assert.PanicTestFunc)(0x104f6b1f0) should not panic
        	            		Panic value:	Invalid name validation scheme requested: unset
        	            		Panic stack:	goroutine 115 [running]:
        	            	runtime/debug.Stack()
        	            		/opt/homebrew/Cellar/go/1.26.1/libexec/src/runtime/debug/stack.go:26 +0x64
        	            	github.com/stretchr/testify/assert.didPanic.func1()
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/stretchr/testify/assert/assertions.go:1286 +0x64
        	            	panic({0x1075532c0?, 0x2e7db7900c60?})
        	            		/opt/homebrew/Cellar/go/1.26.1/libexec/src/runtime/panic.go:860 +0x12c
        	            	github.com/prometheus/common/model.ValidationScheme.IsValidLabelName(0x2e7db6c757c0?, {0x2e7db7908510?, 0x8?})
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/prometheus/common/model/metric.go:203 +0xc0
        	            	github.com/prometheus/prometheus/model/relabel.relabel(0x2e7db751d570, 0x2e7db6fc9a90)
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/prometheus/prometheus/model/relabel/relabel.go:326 +0x3bc
        	            	github.com/prometheus/prometheus/model/relabel.ProcessBuilder(...)
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/prometheus/prometheus/model/relabel/relabel.go:276
        	            	github.com/prometheus/prometheus/notifier.relabelAlerts({0x2e7db6b82980, 0x1, 0x2e7db6fc9b28?}, {{0x0?, 0x2e7db6fc9b48?}}, {0x2e7db6fc9c10, 0x1, 0x104f6b098?})
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/prometheus/prometheus/notifier/alert.go:83 +0x164
        	            	github.com/prometheus/prometheus/notifier.(*Manager).Send(0x2e7db7a9c820, {0x2e7db6fc9c10, 0x1, 0x1})
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/prometheus/prometheus/notifier/manager.go:265 +0xd8
        	            	github.com/grafana/loki/v3/pkg/ruler/base.TestNotifierManagerSendWithAlertRelabelReplaceDoesNotPanic.func1()
        	            		/Users/progers/dev/src/github.com/grafana/loki/pkg/ruler/base/notifier_test.go:478 +0x30
        	            	github.com/stretchr/testify/assert.didPanic(0x10808cd60?)
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/stretchr/testify/assert/assertions.go:1291 +0x68
        	            	github.com/stretchr/testify/assert.NotPanics({0x107d4b8c0, 0x2e7db7942fc8}, 0x2e7db7883758, {0x0, 0x0, 0x0})
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/stretchr/testify/assert/assertions.go:1362 +0x70
        	            	github.com/stretchr/testify/require.NotPanics({0x107d7baf0, 0x2e7db7942fc8}, 0x2e7db7883758, {0x0, 0x0, 0x0})
        	            		/Users/progers/dev/src/github.com/grafana/loki/vendor/github.com/stretchr/testify/require/require.go:1711 +0xa4
        	            	github.com/grafana/loki/v3/pkg/ruler/base.TestNotifierManagerSendWithAlertRelabelReplaceDoesNotPanic(0x2e7db7942fc8)
        	            		/Users/progers/dev/src/github.com/grafana/loki/pkg/ruler/base/notifier_test.go:478 +0x3b4
        	            	testing.tRunner(0x2e7db7942fc8, 0x107d33348)
        	            		/opt/homebrew/Cellar/go/1.26.1/libexec/src/testing/testing.go:2036 +0xc4
        	            	created by testing.(*T).Run in goroutine 1
        	            		/opt/homebrew/Cellar/go/1.26.1/libexec/src/testing/testing.go:2101 +0x3a8
        	Test:       	TestNotifierManagerSendWithAlertRelabelReplaceDoesNotPanic
FAIL
FAIL	github.com/grafana/loki/v3/pkg/ruler/base	0.959s
FAIL

This stack trace is on par with the one seen in this Loki issue.

After the change to notifier.go, the test passes.

Which issue(s) this PR fixes:
Fixes #21368

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Note

Medium Risk
Touches ruler alert notification configuration by forcing MetricNameValidationScheme to UTF8, which could subtly affect label validation/relabeling behavior but is narrowly scoped and covered by new tests.

Overview
Fixes a ruler alert-delivery panic by ensuring the synthetic Prometheus notifier config always sets GlobalConfig.MetricNameValidationScheme to model.UTF8Validation, preventing notifier.ApplyConfig from propagating an unset scheme into alert_relabel_configs.

Updates notifier config unit test expectations accordingly and adds targeted tests that (1) assert the scheme is set and (2) reproduce the previously panicking relabel.Replace + Send path to ensure it no longer panics.

Written by Cursor Bugbot for commit 62e6565. This will update automatically on new commits. Configure here.

@paul1r
paul1r requested a review from a team as a code owner April 3, 2026 18:33
@paul1r paul1r added the type/bug Somehing is not working as expected label Apr 3, 2026
@paul1r
paul1r merged commit a5bd996 into main Apr 3, 2026
89 checks passed
@paul1r
paul1r deleted the paul1r/fix_ruler_panic branch April 3, 2026 19:24
loki-gh-app Bot pushed a commit that referenced this pull request Apr 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport release-3.7.x size/M type/bug Somehing is not working as expected

2 participants