Skip to content

Configdb: unsupported rule format version 0 #5387

Closed
@friedrichg

Description

@friedrichg

Describe the bug
it's not possible to read or modify alertmanager config or rules in configdb
after 5578eeb

cortex operators using configdb cannot upgrade to v1.14.1 or v1.15.2

To Reproduce
Steps to reproduce the behavior:
1.

$ docker run --rm -p8000:80 cortexproject/cortex:master-5578eeb -target=configs -configs.database.uri=memory://
level=info ts=2023-06-05T17:11:21.943969Z caller=main.go:193 msg="Starting Cortex" version="(version=1.13.0, branch=master, revision=5578eeb)"
level=info ts=2023-06-05T17:11:21.944369Z caller=server.go:260 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2023-06-05T17:11:21.948459Z caller=module_service.go:64 msg=initialising module=server
level=info ts=2023-06-05T17:11:21.950176Z caller=module_service.go:64 msg=initialising module=configs
level=info ts=2023-06-05T17:11:21.950239Z caller=cortex.go:400 msg="Cortex started"
level=error ts=2023-06-05T17:11:24.003665Z caller=api.go:186 msg="invalid rules" err="unsupported rule format version 0"
curl -v  -XPOST -d '{"alertmanager_config":"{\"receivers\":[{\"name\":\"empty\"}],\"route\":{\"receiver\":\"empty\"}}","rules_files":{},"template_files":{}}' -H 'X-Scope-OrgID: fake' http://127.0.0.1:8000/api/prom/configs/alertmanager

Returns

< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Mon, 05 Jun 2023 17:11:24 GMT
< Content-Length: 49
<
Invalid rules: unsupported rule format version 0

Expected behavior

Previous commit works fine: c930bd6

$ docker run --rm -p8000:80 cortexproject/cortex:master-c930bd6 -target=configs -configs.database.uri=memory://
level=info ts=2023-06-05T17:16:35.41202Z caller=main.go:193 msg="Starting Cortex" version="(version=1.13.0, branch=master, revision=c930bd6)"
level=info ts=2023-06-05T17:16:35.413964Z caller=server.go:260 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2023-06-05T17:16:35.418405Z caller=module_service.go:64 msg=initialising module=server
level=info ts=2023-06-05T17:16:35.420625Z caller=module_service.go:64 msg=initialising module=configs
level=info ts=2023-06-05T17:16:35.422841Z caller=cortex.go:400 msg="Cortex started"
$ curl -v  -XPOST -d '{"alertmanager_config":"{\"receivers\":[{\"name\":\"empty\"}],\"route\":{\"receiver\":\"empty\"}}","rules_files":{},"template_files":{}}' -H 'X-Scope-OrgID: fake' http://127.0.0.1:8000/api/prom/configs/alertmanager
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> POST /api/prom/configs/alertmanager HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.87.0
> Accept: */*
> X-Scope-OrgID: fake
> Content-Length: 136
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Date: Mon, 05 Jun 2023 17:20:00 GMT
<
* Connection #0 to host 127.0.0.1 left intact

Environment:

  • Infrastructure: laptop but also failing in k8s

Additional Context
Problem is still present in latest cortex

$ docker run --rm -p8000:80 cortexproject/cortex:master-d988472 -target=configs -configs.database.uri=memory://
Unable to find image 'cortexproject/cortex:master-d988472' locally
master-d988472: Pulling from cortexproject/cortex
f56be85fc22e: Already exists
f93ec6642e04: Pull complete
aa4e4b25d019: Pull complete
78bd25eac331: Pull complete
Digest: sha256:8bcbadaaf6e77107fad6fcb5f8a2d16c2240c78ba4aadf464f14759456a1fe7f
Status: Downloaded newer image for cortexproject/cortex:master-d988472
ts=2023-06-05T17:24:04.779773Z caller=main.go:201 level=info msg="Starting Cortex" version="(version=1.15.1, branch=master, revision=d988472)"
ts=2023-06-05T17:24:04.781333Z caller=server.go:323 level=info http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
ts=2023-06-05T17:24:04.786709Z caller=module_service.go:64 level=info msg=initialising module=server
ts=2023-06-05T17:24:04.790977Z caller=module_service.go:64 level=info msg=initialising module=configs
ts=2023-06-05T17:24:04.791389Z caller=cortex.go:422 level=info msg="Cortex started"
ts=2023-06-05T17:24:16.082692Z caller=api.go:186 level=error msg="invalid rules" err="unsupported rule format version 0"

The following simple form fails too

curl -v  -d '{}' -H 'X-Scope-OrgID: fake' http://127.0.0.1:8000/api/prom/configs/alertmanager

Metadata

Metadata

Assignees

No one assigned

    Labels

    component/rulesBits & bobs todo with rules and alerts: the ruler, config service etc.type/bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions