ARC 0.13.x controller log errors ephemeral runner failed to create pod resource then reconcile error (context deadline exceeded)

[durangogt]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

ARC (Actions Runner Controller)

Discussion Details

TL;DR: ARC 0.13.x controller fails to create ephemeral runner pods on
Karpenter-managed EKS with "context deadline exceeded" errors. Working with
bare minimal values but can't add nodeSelector/ephemeral volumes. Pod creation
times out before Karpenter can provision nodes.

What's failing: Pod creation at ephemeralrunner_controller.go:687
When: During reconciliation loop
Impact: Runners never come online despite valid GitHub token

Already Verified (Not the Issue)

• GitHub token is valid (runners appear offline in GitHub)
• Karpenter works (test pods with nodeSelector succeed)
• ARC controller CRDs installed correctly
• Both controller and runner-set namespaces exist
• K8s 1.34, Helm 3.17.1 are compatible versions

Questions for Community

1. Has anyone successfully used ARC 0.13.0+ with AWS EKS + Karpenter and ephemeral runners?
2. Is there a known issue with volumeClaimTemplate creation timing in 0.13.x?
3. Should we revert to the legacy runner controller?
4. Is syncPeriod=1m too aggressive for Karpenter node provisioning?

We are migrating from public GitHub runners to private runners mainly due to GitHub Copilot Coding Agent needs private connectivity for us.

Steps to reproduce error of "...Timeout: request did not complete within requested timeout - context deadline exceeded..." (see full log at bottom):

1. Install controller: (same error seen when doing latest version 0.13.1)
   helm install arc --namespace "arc-systems" --version 0.13.0 oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

2. Install arc-runner-set: (same error seen when doing latest version 0.13.1)
   helm install "arc-runner-set" --namespace "github-runners" --version 0.13.0 -f values-minimal.yml oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
   values-minimal.yml looks like:

githubConfigUrl: "https://github.com/<our-org>/<repo>" # repo level self-hosted runner for now
githubConfigSecret: "github-token-secret"
maxRunners: 10
minRunners: 2

runnerScaleSetName: "terraform-runners"

runnerGroup:  "Default"

template:
  spec:
    serviceAccountName: github-runner-sa

    nodeSelector: # This ensures its running on Karpenter nodes
      managedBy: karpenter
      nodepool: default-pool
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
        - name: ACTIONS_RUNNER_CONTAINER_HOOKS
          value: /home/runner/k8s/index.js
        - name: ACTIONS_RUNNER_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "true"
        # ADD: Configure ephemeral mode
        - name: RUNNER_EPHEMERAL
          value: "true"  # ← Runner terminates after each job
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "local-path"
              resources:
                requests:
                  storage: 50Gi

3. When running the above we get the main error logs on the controller as shown at the bottom BUT also never see in Karpenter logs or nodeclaims any activity to bring up nodes. However when running a test pod with node selector set to Karpenter managed nodes (just as above), the test pod comes up fine. So I know Karpenter is working fine.

4. Even w/ the above Helm values and the nodeSelector commented out so that we take Karpenter out of play & just have the runner pods run on managed nodes but it still fails with the same error below.

5. At this point the only thing I can get running consistently is the below extreme bare, no karpenter nodes values & it just runs on the managed node group that Karpenter runs on (not what we want):

githubConfigUrl: "https://github.com/<org>/<repo>"
githubConfigSecret: "github-token-secret"
maxRunners: 10
minRunners: 2

Error log

│ 2026-01-16T16:56:43Z    INFO    EphemeralRunner    Created new pod spec for ephemeral runner    {"version": "0.13.0", "ephemeralrunner": {"name":"terraform-runners-qdkp5-runner-rqq7c", │
│ "namespace":"github-runners"}}                                                                                                                                                           │
│ 2026-01-16T16:57:17Z    ERROR    EphemeralRunner    Failed to create pod resource for ephemeral runner.    {"version": "0.13.0", "ephemeralrunner": {"name":"terraform-runners-qdkp5-run │
│ ner-rqq7c","namespace":"github-runners"}, "error": "Timeout: request did not complete within requested timeout - context deadline exceeded"}                                             │
│ github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*EphemeralRunnerReconciler).createPod                                                                   │
│     github.com/actions/actions-runner-controller/controllers/actions.github.com/ephemeralrunner_controller.go:687                                                                        │
│ github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*EphemeralRunnerReconciler).Reconcile                                                                   │
│     github.com/actions/actions-runner-controller/controllers/actions.github.com/ephemeralrunner_controller.go:278                                                                        │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile                                                                                                      │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:216                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler                                                                                               │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:461                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem                                                                                            │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1                                                                                                  │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296                                                                                                     │
│ 2026-01-16T16:57:17Z    ERROR    EphemeralRunner    Failed to create the pod    {"version": "0.13.0", "ephemeralrunner": {"name":"terraform-runners-qdkp5-runner-rqq7c","namespace":"git │
│ hub-runners"}, "error": "Timeout: request did not complete within requested timeout - context deadline exceeded"}                                                                        │
│ github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*EphemeralRunnerReconciler).Reconcile                                                                   │
│     github.com/actions/actions-runner-controller/controllers/actions.github.com/ephemeralrunner_controller.go:294                                                                        │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile                                                                                                      │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:216                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler                                                                                               │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:461                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem                                                                                            │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1                                                                                                  │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296                                                                                                     │
│ 2026-01-16T16:57:17Z    ERROR    Reconciler error    {"controller": "ephemeralrunner", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "EphemeralRunner":  │
│ {"name":"terraform-runners-qdkp5-runner-rqq7c","namespace":"github-runners"}, "namespace": "github-runners", "name": "terraform-runners-qdkp5-runner-rqq7c", "reconcileID": "f9f7b28f-1c │
│ 74-413a-a7fa-170805ba6a7b", "error": "Timeout: request did not complete within requested timeout - context deadline exceeded"}                                                           │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler                                                                                               │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:474                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem                                                                                            │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421                                                                                                     │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1                                                                                                  │
│     sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296                                                                                                     │
│ 2026-01-16T16:57:17Z    INFO    EphemeralRunner    Ephemeral runner pod does not exist. Creating new ephemeral runner    {"version": "0.13.0", "ephemeralrunner": {"name":"terraform-run │
│ ners-qdkp5-runner-rqq7c","namespace":"github-runners"}}                                                                                                                                  │
│ 2026-01-16T16:57:17Z    INFO    EphemeralRunner    Creating new pod for ephemeral runner    {"version": "0.13.0", "ephemeralrunner": {"name":"terraform-runners-qdkp5-runner-rqq7c","nam │
│ espace":"github-runners"}}                                                                                                                                                               │
│ 2026-01-16T16:57:17Z    INFO    EphemeralRunner    Created new pod spec for ephemeral runner    {"version": "0.13.0", "ephemeralrunner": {"name":"terraform-runners-qdkp5-runner-rqq7c", │
│ "namespace":"github-runners"}}       

Background of goal:
We run Karpenter on AWS EKS clusters quite a bit to utilize AWS EC2 SPOT nodes for our workloads. Now we are slowly migrating from Azure Devops agent runners on AWS ECS to github runners on AWS EKS. This goal is mainly for running Github Copilot Coding Agent on this private/self-hosted runner. We have been using the Coding Agent a lot on the public runner but need to private connectivity.

Ideally we would like to install all the tools and settings we have in our usual Docker container into the actions/runner-image then launch it according to the ARC runner Helm values examples. We have semi successfully done this with Karpenter to github runners but when upgrading the controller & scale sets to 0.13.1 there were some problems (possibly conflicting CRDs?) and we had to start over; ever since then we can't reproduce even the semi successful state due to the error at the bottom of this post. So we've reduced the runner set values to very very minimal to see what will work but are hitting this wall. Any help would be very much appreciated!

It seems like the ARC controller isn't waiting enough time to allow Karpenter to bring up a node & then allocate the runner to it, but we aren't sure about that. Should we try the actions-runner-controller legacy version instead w/ an increase to the syncPeriod to something greater than 1 minute? The runner-listener logs are all normal it seems but we just can't figure out what's going on with this context deadline exceeded error.

[syedabdulbasitali1]

Hi there, I've run into this exact "context deadline exceeded" wall before with ARC on EKS. It is incredibly frustrating because it fails silently.

The Root Cause The reason Karpenter isn't provisioning nodes is that the Pods aren't actually being created. The error happens during the API Admission phase.

Basically, the K8s API server receives the request to create the runner pod, tries to send it to a MutatingWebhook (likely ARC's own webhook) to inject the container hooks, and that connection is timing out. Because the webhook fails, the API server rejects the Pod creation entirely. The Pod never hits the Pending state, so Karpenter never sees it.

Since you mentioned migrating/upgrading versions, this is almost certainly one of two things:

1. "Ghost" Webhooks (Most Likely) If you have any leftover MutatingWebhookConfiguration resources from a previous install (especially if you moved from Legacy ARC -> Scale Sets or 0.13.0 -> 0.13.1), the API server might be trying to send requests to a service that no longer exists.

Check: Run kubectl get mutatingwebhookconfigurations.

Fix: Delete any webhooks that look old or related to the "summerwind" legacy controller. Only the one matching your current active deployment should remain.

2. The EKS Control Plane Firewall The Kubernetes API server (managed by AWS) needs to talk to your ARC Controller Pod to mutate the runner pod.

The Issue: If your ARC Controller is running on a private node, the EKS Control Plane Security Group might not have permission to talk to your Node Security Group on port 9443.

Fix: Ensure your Node Security Group allows Inbound traffic from the EKS Cluster Security Group on port 9443.

One quick test: Try disabling the volumes and volumeMounts in your runner spec for just one run. Sometimes (rarely) the storage driver's webhook is the one timing out. But 90% of the time, it's the ARC webhook connection failing.

Don't revert to Legacy yet! This is just a network/webhook config block. Once you clear that webhook timeout, the pod will go Pending and Karpenter will snap it up immediately.

[durangogt]

Thank you very much! This was exactly the problem, as soon as I cleared out the left over mutating webhooks it worked just as you explained.

Now I'm working out why the runners keep trying to join GitHub sessions that are already established or not ending them cleanly, not sure which one is the root cause.

[durangogt]

And now with the new release of scale set client (without k8s) I'm wondering if I should try that out here soon

https://github.blog/changelog/2026-02-05-github-actions-early-february-2026-updates/

[sitta07]

This looks like an ARC 0.13.x regression with ephemeral runners: the controller times out creating the runner pod (context deadline exceeded) before Karpenter or volume provisioning can complete. It’s not your token, Karpenter, or cluster version. Current workarounds are reverting to the legacy controller / older ARC version or removing ephemeral volumes and advanced pod specs. This should be reported upstream so ARC waits properly for pod + PVC creation on Karpenter-backed clusters 🔥

[mislavperi]

I've had this happen if a node is acting weird, by draining a node I resolved it. I run ARC 0.13.1 on EKS 1.32