Skip to content

fix(helm): create namespaced RBAC when sidecar is enabled - #16776

Merged
Jayclifford345 merged 5 commits into
grafana:mainfrom
toVersus:fix/rbac-namespaced
Jul 29, 2025
Merged

fix(helm): create namespaced RBAC when sidecar is enabled#16776
Jayclifford345 merged 5 commits into
grafana:mainfrom
toVersus:fix/rbac-namespaced

Conversation

@toVersus

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

When RBAC is namespaced, the necessary permissions for Secrets and ConfigMaps are not granted even when the sidecar is enabled, resulting in the permission error.

loki-0 loki-sc-rules {"time": "2025-03-17T05:08:41.358310+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '37517204-4893-43ef-ab3a-d49607f24d8d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '85650ad3-58f1-424a-a80d-428ed6839376', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f53e282a-4d15-4681-be6f-5ebce100ee99', 'Date': 'Mon, 17 Mar 2025 05:08:41 GMT', 'Content-Length': '304'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"configmaps is forbidden: User \\\\\"system:serviceaccount:monitoring:loki\\\\\" cannot watch resource \\\\\"configmaps\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"monitoring\\\\\"\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"configmaps\"},\"code\":403}\\n'\n\n", "level": "ERROR"}

In the current Helm chart logic, the generation of RBAC for the sidecar depends on OpenShift’s Security Context Constraints. As a result, the required RBAC is not created in non-OpenShift environments.

To address this, we will modify the RBAC rule generation logic to ensure that the necessary Role and RoleBinding are created when the sidecar is enabled and RBAC is namespaced.

Which issue(s) this PR fixes:
Fixes #15845

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR
@toVersus
toVersus requested a review from a team as a code owner March 17, 2025 05:12
@CLAassistant

CLAassistant commented Mar 17, 2025

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@QuentinBisson

QuentinBisson commented Jul 23, 2025

Copy link
Copy Markdown
Contributor

Hey @toVersus. Thank you for contributing.

Do you mind adding a changelog entry for your fix as well?

@toVersus
toVersus force-pushed the fix/rbac-namespaced branch from 93d0afb to f5f4d03 Compare July 23, 2025 08:57
@toVersus

Copy link
Copy Markdown
Contributor Author

@QuentinBisson
Thanks for taking the time to look into this! I've added a changelog entry.

@QuentinBisson

Copy link
Copy Markdown
Contributor

LGTM @Jayclifford345 :)

@toVersus
toVersus force-pushed the fix/rbac-namespaced branch 3 times, most recently from ac89c9c to 119bd35 Compare July 24, 2025 02:43
@Jayclifford345

Copy link
Copy Markdown
Contributor

Amazing, thank you @QuentinBisson, if all pass, then we can get this merged. Thank you for contributing this @toVersus

When RBAC is namespaced, the necessary permissions for Secrets
and ConfigMaps are not granted even when the sidecar is enabled,
resulting in the permission error. In the current Helm chart logic,
the generation of RBAC for the sidecar depends on OpenShift’s
Security Context Constraints. As a result, the required RBAC is not
created in non-OpenShift environments.

To address this, we will modify the RBAC rule generation logic to
ensure that the necessary Role and RoleBinding are created
when the sidecar is enabled and RBAC is namespaced.

Signed-off-by: Tsubasa Nagasawa <toversus2357@gmail.com>
@toVersus
toVersus force-pushed the fix/rbac-namespaced branch from 1027906 to c53c519 Compare July 25, 2025 01:45
JStickler and others added 2 commits July 28, 2025 09:23
Signed-off-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com>
@Jayclifford345
Jayclifford345 enabled auto-merge (squash) July 29, 2025 09:32
Signed-off-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com>
Signed-off-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com>
@Jayclifford345
Jayclifford345 merged commit 7c9003e into grafana:main Jul 29, 2025
74 checks passed
@toVersus
toVersus deleted the fix/rbac-namespaced branch July 29, 2025 23:16

@rajgopal-a rajgopal-a left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’re running into a problem with the chart when using rbac.namespaced=true along with sidecar.rules.enabled=true. The error message is "Failed to create resource: roles.rbac.authorization.k8s.io.s is forbidden. User "system:service:account:" cannot create resources roles in API group "rbac.authorization.k8s.io" in the namespace . As per our internal policy, the service account that runs Helm upgrades does not have permission to create or modify Roles or RoleBindings

Right now, with this configuration, the chart always tries to create a Role, which causes the deployment to fail. We’d like to continue using namespaced RBAC with the sidecar enabled, but avoid Helm failing because of RBAC restrictions.

Could you advise on options here?

Is there a supported way to tell the chart not to manage RBAC objects.
If not today, would you consider adding a flag like rbac.create: false (or similar) to skip generating Roles/RoleBindings with this configuration?
Any documented workaround you recommend until then?

@toVersus

Copy link
Copy Markdown
Contributor Author

I fixed a bug where RBAC was not created outside of OpenShift when RBAC is namespaced. Since it seems to be an edge case for an organization to be unable to create RBAC with their policies, I think we need to handle the addition of options separately. Currently, there is no workaround, so we have no choice but to use an older version of the Helm chart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment