Github Actions Runner pods gets deleted as soon as it comes up

[meetatmoe]

I've configured Github Action controller using official guide.

When the job is triggered, runner pod gets created but as soon as pod gets initialized, it gets deleted.
Here are the error logs before pod gets deleted.

runner √ Connected to GitHub
runner [RUNNER 2024-05-02 08:56:30Z INFO Terminal] WRITE LINE: 
runner 
runner [RUNNER 2024-05-02 08:56:31Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
runner [RUNNER 2024-05-02 08:56:31Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
runner [RUNNER 2024-05-02 08:56:31Z ERR  GitHubActionsService] POST request to https://pipelinesghubeus9.actions.githubusercontent.com/REDACTED/_apis/oauth2/token failed. HTTP Status: BadRequest
runner [RUNNER 2024-05-02 08:56:31Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
runner [RUNNER 2024-05-02 08:56:31Z ERR  MessageListener] Catch exception during create session.
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener] GitHub.Services.OAuth.VssOAuthTokenRequestException: Registration a87a388f-7975-4edd-ae39-51647b21f3e0 was not found.
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Runner.Listener.MessageListener.CreateSessionAsync(CancellationToken token)
runner [RUNNER 2024-05-02 08:56:32Z ERR  Terminal] WRITE ERROR: Failed to create a session. The runner registration has been deleted from the server, please re-configure. Runner registrations are automatically deleted for runners that have not connected to the service recently.
runner Failed to create a session. The runner registration has been deleted from the server, please re-configure. Runner registrations are automatically deleted for runners that have not connected to the service recently.
runner [RUNNER 2024-05-02 08:56:32Z INFO Listener] Runner execution has finished with return code 1
runner Runner listener exit with terminated error, stop the service, no retry needed.
runner Exiting runner...
dind time="2024-05-02T08:56:32.030184965Z" level=info msg="Processing signal 'terminated'"
dind time="2024-05-02T08:56:32.031672419Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
dind time="2024-05-02T08:56:32.037650221Z" level=info msg="Daemon shutdown complete"
dind time="2024-05-02T08:56:32.037712487Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
dind time="2024-05-02T08:56:32.037779240Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby

[snimesh17]

facing exact same issue.

any solution you found?

[pulkitanz]

I am still seeing this happening across a lot of our pods very intermittently. Another issue is that once the pod is deleted then the Ephemeral Runner Resource is also not cleaned up.

[rp-martinbarmaymon]

Has anyone found a fix or workaround for this? We started experiencing this a few weeks ago and the issue is intermittent. Some days we have no errors, and on other days, we see errors in 20-30% of our pods. These errors are concentrated in short bursts, all occurring within a few minutes.

[andrey-petrenko-develeap]

How do you deal with it anyway when you encounter it?
For us last time the only thing that seemed to help is force reinstalling the scale set helm chart

[rp-martinbarmaymon]

Hi @andrey-petrenko-develeap, hanks for the reply! So far, we haven't found any other workaround to deal with it; Some runner pods fail with that error and ARC still count it as running, but they are not registered on Github. In this scenario the job remains in pending state until a dev cancels the workflow and re-runs it, hoping for the newer runner to work properly.
I'm going to try your workaround, as I'm still facing errors. I've just deleted and reinstalled all my scale sets to see if that helps and I'll report back here in a few days once I have more metrics on whether it worked or not.

[rp-martinbarmaymon]

Just reporting back, we removed and reinstalled our ARC stack and we still have occurrences of this issue

[rp-martinbarmaymon]

If anyone is still facing this issue, we didn't find how to solve the issue, but we've found a workaround: setting a minimum number of runners. This is not optimal or efficient, but it patches the problem.
In our scenario, when a new workflow was triggered, ARC would start a new runner pod and then move on, without checking if it was actually running. This meant our Action Jobs would get stuck in pending because there was no available runner when a new one failed. Having two permanent runners "solved" that issue because if the new runner failed, there was still a backup runner that would eventually be available.

To track which jobs were stuck in pending, we had to create a custom script to calculate this metric and send it to New Relic, as GitHub does not offer it natively. This is the script we are using, which we execute as a cron job. It gets a list of repositories to monitor (in our case, only a few), checks the jobs that are in a pending state, calculates how long they have been pending, and if the pending time is greater than 3 minutes, it sends that job's data to New Relic.

#!/bin/bash
# Required environment variables:
# - GITHUB_TOKEN: GitHub personal access token with access to all repos required
# - NEW_RELIC_API_KEY: New Relic insert key  
# - NEW_RELIC_ACCOUNT_ID: New Relic account ID

ORG_NAME="your_organization_name"
REPO_NAMES=("repository_1" "repository_2" "repository_n")
NEW_RELIC_EVENTS_URL="https://insights-collector.newrelic.com/v1/accounts/${NEW_RELIC_ACCOUNT_ID}/events"
# Function to log messages (only if VERBOSE is set)
log() {
    if [ "${LOG_VERBOSE:-0}" = "1" ]; then
        echo "$1" >&2
    fi
}
calculate_pending_duration() {
    local started_at=$1
    export TZ=UTC
    local current_time=$(date -u +%s)
    local started_time=$(date -d "$started_at" +%s 2>/dev/null || date -j -f "%Y-%m-%dT%H:%M:%SZ" "$started_at" +%s 2>/dev/null)
    if [ -n "$started_time" ]; then
        echo $((current_time - started_time))
    else
        echo "0"
    fi
    unset TZ
}
# Function to create New Relic event JSON
create_newrelic_event() {
    local repo=$1 workflow_name=$2 run_id=$3 workflow_url=$4 job_id=$5
    local job_name=$6 job_status=$7 job_url=$8 started_at=$9 pending_seconds=${10} workflow_status=${11} workflow_id=${12}
    jq -n \
        --arg eventType "Pipeline" \
        --arg timestamp "$(date -u +%s)" \
        --arg repository "$repo" \
        --arg organization "$ORG_NAME" \
        --arg workflow_name "$workflow_name" \
        --arg workflow_id "$workflow_id" \
        --arg job_id "$job_id" \
        --arg job_name "$job_name" \
        --arg job_status "$job_status" \
        --arg job_url "$job_url" \
        --argjson pending_duration_seconds "$pending_seconds" \
        --arg workflow_status "$workflow_status" \
        '{
            eventType: $eventType,
            repository: $repository,
            workflow_name: $workflow_name,
            workflow_id: $workflow_id,
            job_id: $job_id,
            job_name: $job_name,
            job_status: $job_status,
            job_url: $job_url,
            pending_duration_seconds: $pending_duration_seconds,
            workflow_status: $workflow_status
        }'
}
# Function to send event to New Relic
send_to_newrelic() {
    local event_json="$1"
    local response=$(curl -s -w "\n%{http_code}" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-Insert-Key: $NEW_RELIC_API_KEY" \
        -d "$event_json" \
        "$NEW_RELIC_EVENTS_URL")
    local http_code=$(echo "$response" | tail -n1)
    local response_body=$(echo "$response" | sed '$d')  # Remove last line
    if [ "$http_code" = "200" ] || [ "$http_code" = "202" ]; then
        log "  Event sent to New Relic successfully"
    else
        log "  Failed to send to New Relic (HTTP $http_code): $response_body"
        echo "$event_json"  # Fallback: output to stdout
    fi
}
# Function to process pending jobs for a workflow run
process_workflow_jobs() {
    local repo=$1 run_id=$2 workflow_name=$3 workflow_url=$4 workflow_status=$5 workflow_id=$6
    local jobs_response=$(curl -s -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github+json" "https://api.github.com/repos/${ORG_NAME}/${repo}/actions/runs/${run_id}/jobs")
    echo "$jobs_response" | jq -r '
        .jobs[]? |
        select(.status == "queued" or .status == "pending") |
        "\(.id)|\(.name)|\(.started_at // "null")|\(.html_url)|\(.status)"
    ' | while IFS='|' read -r job_id job_name started_at job_url job_status; do
        [ -z "$job_id" ] || [ "$job_id" = "null" ] && continue
        # Use started_at or current time as fallback
        local time_reference="${started_at}"
        [ "$time_reference" = "null" ] && time_reference=$(date -u -Iseconds)
        local pending_seconds=$(calculate_pending_duration "$time_reference")
        # Only output if pending longer than threshold
        if [ "$pending_seconds" -gt 180 ]; then
            log "  → Sending event for job: $job_name (${pending_seconds}s pending)"
            local event_json=$(create_newrelic_event "$repo" "$workflow_name" "$run_id" "$workflow_url" \
                "$job_id" "$job_name" "$job_status" "$job_url" "$time_reference" \
                "$pending_seconds" "$workflow_status" "$workflow_id")
            send_to_newrelic "$event_json"
        fi
    done
}
# Function to process workflow runs for a repository
process_repository_workflows() {
    local repo=$1
    log "Checking repository: $repo"
    for status in "queued" "in_progress"; do
        local workflow_runs=$(curl -s -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github+json" "https://api.github.com/repos/${ORG_NAME}/${repo}/actions/runs?status=${status}")
        local workflow_count=$(echo "$workflow_runs" | jq -r '.workflow_runs | length')
        if [ "$workflow_count" -gt 0 ]; then
            log "  Found $workflow_count $status workflows"
        fi
        echo "$workflow_runs" | jq -r '
            .workflow_runs[]? |
            "\(.id)|\(.name)|\(.html_url)|\(.workflow_id)"
        ' | while IFS='|' read -r run_id workflow_name workflow_url workflow_id; do
            [ -z "$run_id" ] || [ "$run_id" = "null" ] && continue
            process_workflow_jobs "$repo" "$run_id" "$workflow_name" "$workflow_url" "$status" "$workflow_id"
        done
    done
    local events_count=$(echo "$workflow_runs" | jq -r '
        [.workflow_runs[]? | select(.id != null)] | length
    ' 2>/dev/null || echo "0")
    if [ "$events_count" -eq 0 ]; then
        log "  No pending jobs found" >&2
    fi
}
for repo in "${REPO_NAMES[@]}"; do
    process_repository_workflows "$repo"
done

[melloc01]

We're seeing this on GHES after upgrade to 3.16.2 -> 3.16.9. Support says we need to upgrade ARC so we're trying this now.
Have you come to any conclusion?

May I as what's your ARC version?