Runners not scaling when EphemeralRunner status is Failed

[bryanjuho]

When EphemeralRunners status is marked as Failed, runners fail to scale to the minimum (3 in my case) number.

I guess Runners tend to fail every once in a while, but since they do not shutdown automatically but still counts as a runner, the maximum runner count is reached. These holds all queued jobs.

According to the runner-set-listener logs, replicas count shows -1 when all three of my runners had failed.

{"namespace": "devops", "name": "runner-set-n6bk6", "replicas": -1}

Any ideas? Thanks!

[DamianPalyskaSpotOn]

I had the same problem. The solution was to bump the runner image version to the latest.

[bryanjuho]

Unfortunately it did not, I ended up creating a CronJob resource to delete failed runners periodically.

[krupakar1329]

`2024-05-16T15:50:01Z INFO EphemeralRunner EphemeralRunner has failed more than 5 times. Marking it as failed {"ephemeralrunner": {"name":"pd-cluster-znz6x-runner-4ctcc","namespace":"github-arc-runner-xxx"}}
2024-05-16T15:50:01Z INFO EphemeralRunner Updating ephemeral runner status to Failed {"ephemeralrunner": {"name":"pd-cluster-znz6x-runner-4ctcc","namespace":"github-arc-runner-x"}}xx
2024-05-16T15:50:01Z INFO EphemeralRunner Removing the runner from the service {"ephemeralrunner": {"name":"pd-cluster-znz6x-runner-4ctcc","namespace":"github-arc-runner-xxx"}}
2024-05-16T15:50:01Z INFO EphemeralRunnerSet Ephemeral runner counts {"ephemeralrunnerset": {"name":"pd-cluster-znz6x","namespace":"github-arc-runner-x"}, "pending"xx: 0, "running": 0, "finished": 0, "failed": 4, "deleting": 0}
2024-05-16T15:50:01Z INFO EphemeralRunnerSet Scaling comparison {"ephemeralrunnerset": {"name":"pd-cluster-znz6x","namespace":"github-arc-runner-xxx"}, "current": 4, "desired": 4}
2024-05-16T15:50:01Z INFO EphemeralRunnerSet Updating status with current runners count {"ephemeralrunnerset": {"name":"pd-cluster-znz6x","namespace":"github-arc-runner-xxx"}, "count": 4}

runners are not seen in giithub , runner pods are getting deleted. any clue
`

[bryanjuho]

I was never able to find out why they fail, but since they autoscale, I just delete them.

[alexgaganashvili]

After trying to figure out why new runners do not get scheduled, I found out that that Failed ephemeralrunners were to blame. I opened a new discussion thread yesterday. I started thinking of creating a cron job and see in this post a confirmation that that's what I'll most likely have to do.
Btw, what's interesting is that I'm not running into this issue on a different cluster. So I'm wondering what the root cause it.
Whatever it is, it would be great if Failed ephemeralrunners were automatically cleaned up.
fyi: @nikola-jokic

[koraypinarci]

I have the same issue, and i created a cron-job that cleans up all failed ephemeralrunners so that the jobs can continue running. It would be better, of course, if ARC could recognize and clean them up by itself.

[kiranbayram]

Hello there @bryanjuho , could you please provide the details how exactly you set up the CronJob and what commands you're running in it? Thanks in advance. Regards 🙏

[bryanjuho]

Hi, I'm sure there's a better way to do this but this is how I've done it.
hope this helps

The Cronjob resource

# CronJob to delete failed runners
apiVersion: batch/v1
kind: CronJob
metadata:
  name: delete-failed-ephemeral-runners
  namespace: development
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: runner-manager
          containers:
          - name: delete-failed-runners
            image: my-registry/delete-runners:latest
            imagePullPolicy: Always
          restartPolicy: OnFailure

The Dockerfile for the delete-runners image

FROM debian:bookworm-slim

# Install kubectl
RUN apt-get update && apt-get install -y curl \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

# Add your script to the container
WORKDIR /app
COPY delete-failed-runners.sh .

RUN chmod +x delete-failed-runners.sh

ENTRYPOINT ["/app/delete-failed-runners.sh"]

Lastly the shell script (delete-failed-runners.sh)

#!/bin/bash

# Namespace where your EphemeralRunners are deployed
NAMESPACE="development"

# Get the list of EphemeralRunner resources with status 'Failed'
failed_runners=$(kubectl get EphemeralRunner -n $NAMESPACE --no-headers -o custom-columns=":metadata.name,:status.phase" | grep Failed | awk '{print $1}')

for runner in $failed_runners; do
    echo "Deleting failed EphemeralRunner: $runner"
    kubectl delete EphemeralRunner $runner -n $NAMESPACE
done

[kiranbayram]

This is great ��� thanks so much 🙏 @bryanjuho I will look into it

[krupakar1329]

Seeing the issue again Found below in pod logs. 2.317.0 version is deprecated

[RUNNER 2024-09-17 09:15:23Z INFO Terminal] WRITE LINE: Current runner version: '2.317.0'
Current runner version: '2.317.0'
[RUNNER 2024-09-17 09:15:23Z INFO Terminal] WRITE LINE: 2024-09-17 09:15:23Z: Listening for Jobs
2024-09-17 09:15:23Z: Listening for Jobs
[RUNNER 2024-09-17 09:15:23Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-09-17 09:15:23Z ERR  GitHubActionsService] GET request to https://pipelinesghubeus4.actions.githubusercontent.com/GcLiJYpRBu4UULaLKeoKHekWwfyzWN1jFTCRGiRwdwkcnwbFRH/_apis/distributedtask/pools/5/messages?sessionId=d795c87b-01b9-4d44-988a-ad7ebf46b674&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-09-17 09:15:23Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-09-17 09:15:23Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-09-17 09:15:23Z ERR  Terminal] WRITE ERROR: An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-17 09:15:23Z ERR  Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.RunAsync(RunnerSettings settings, Boolean runOnce)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Runner.ExecuteCommand(CommandSettings command)
[RUNNER 2024-09-17 09:15:23Z ERR  Listener]    at GitHub.Runner.Listener.Program.MainAsync(IHostContext context, String[] args)
Runner listener exit with terminated error, stop the service, no retry needed.

[medounesa]

i have same issue after update to the last version 2.319.1 but the ephemeral runner still offline

[postsa]

+1 hit this issue as well. I was surprised by the behavior. Successful jobs are cleared, but failed jobs eat into my total job count indefinitely. If there's an ephemeral error with networking in the cluster that causes multiple jobs to fail my runner set is essentially offline because all jobs are stuck in the queue. We could write a cron job to handle this as others have, but it's unexpected behavior IMO.

[woojae-highnote]

I'm getting similar symptoms. There seems to be some "ghost" runners that hangs around in a queue doing nothing. One way I mitigated the issue was to increase minRunners.

[arouxAgicap]

We also encounter this issue at the moment with the last version of the image runner and the last version of the helm chart. It started out of the blue on monday at 10:40 AM CEST.
We manage to mitigate it for a day but It happens again today at the same time. We planned to implement the cronjob to mitigate the issue durably but we are extremely frustrated since we also found this log and we have no answer from GitHub support

[RUNNER 2025-04-02 09:55:35Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=*************&status=Online&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: NotFound

[matthew-collins-dailypay]

+1 We are also encountering this issue, it seems to have started ~March 28th for us.

[cmiller01]

Another +1, happened April 2. I have 3 separate scale sets running and all were impacted at the same time. Clearing the Failed ephemeralrunners allowed the jobs to pickup.

[woojae-highnote]

This started on March 28th across multiple clusters, and it only happens between 2:00am and 4:00am PDT. Before the 28th we rarely had a failed job.

[avadhanij]

The timing is too coincidental. It started happening to us from the first week of April and this has become a real pain.

We have been considering the idea of a cronjob that cleans up the failed runners as well, but I have noticed that there are some runners that just won't go away, as if it was stuck on some finalizer. There is no pod associated with the runner, so it's not clear why it isn't going away.

Anyone else experience the same?

[oliverpark999]

Wow, I had the same problem.
actions/actions-runner-controller#4073 (comment)

[dududko]

Happened for us on June 7 with 7 runners being stuck in the Failed state. Manual deletion of ephemeralrunners helped to resolve the issue. Will try to investigate further on what exactly cause the problem.

[michel-numan]

Did you ever find out @dududko ?

[dududko]

@michel-numan Kinda
Long story short:

• 30 days after a new version of GH runner is available the old version becomes deprecated: link
• For self-hosted on-premises runners this is not an issue since they are automatically updated to the newest version
• For ephemeral runners this is not an issue if you use a default docker image for a runner container with a latest tag
• It is a problem though if you modify a runner image to include some of your own tooling into it!!! I this case when a new version is released you would have to create a new version of your docker image to use latest runner version.

So the best thing to do is to use default runner image with latest tag and then inside your workflows specify in which image the job should run and inside that image put all the tooling you need. By doing this you eliminate the problem. (I think it is also possible to enable auto-update, but then your runner will always have to download a new version once the old one is deprecated; with proposed solution it is done only once and then docker image is cached).

For us it would take some time to make this change so I increased observability and monitoring so we know when this problem repeats and can act quickly.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬