Skip to content

fix(auto-approve-bot-prs): hold a failed check while its rerun is in flight - #248

Merged
sydorovdmytro merged 3 commits into
mainfrom
devops-1454/auto-approve-failure-hold
Sep 1, 2026
Merged

fix(auto-approve-bot-prs): hold a failed check while its rerun is in flight#248
sydorovdmytro merged 3 commits into
mainfrom
devops-1454/auto-approve-failure-hold

Conversation

@sydorovdmytro

Copy link
Copy Markdown
Contributor

Closes DEVOPS-1454

Nothing re-arms this action: it triggers on pull_request opened/synchronize,
and re-running a check is neither, so the first red poll was the last word even
when the rerun went green minutes later. The dedup ranking already resolves
failure then success -> success; it just never got a second look.

On vcluster-pro#2367 one flaky spec out of 190 failed at 22:47, the bot gave up
at 22:47:42, the rerun passed at 00:54, and the release cut waiting on that merge
had already timed out. Third time this shape has stalled a cut (DEVOPS-1254,
vcluster-pro#2155).

Reuses the newer-check-suite watermark the cancelled path already has. A
failure is held only while a newer suite is actually running, so a genuinely
broken PR still bails on the first poll and nothing burns wait-max-attempts.
Commit statuses are excluded: no suite id, no rerun concept.

The refusal is now an ::error::. As a ::notice:: under continue-on-error
the job reported success while refusing to merge, which is what made all three
incidents slow to diagnose.

This covers the window where the rerun is already running when we poll. Re-arming
on check_suite: completed would cover the rest but needs the action to resolve
the PR from an event that has no pull_request context; noted as follow-up on
the issue.

Test plan

  • make test-auto-approve-bot-prs — 132 pass, 0 fail (5 new)
  • shellcheck src/wait-for-ci.sh — clean
…flight

Nothing re-arms this action: it triggers on pull_request opened/synchronize, and
re-running a check is neither, so the first red poll was the last word even when
the rerun went green minutes later. One flaky spec out of 190 was enough to
refuse a merge permanently and stall the release cut waiting on it.

Reuse the newer-check-suite watermark the cancelled path already has. A failure
is held only while a newer suite is actually running, so a genuinely broken PR
still bails on the first poll. Commit statuses are excluded: no suite id, no
rerun concept.

The refusal is now an ::error::. As a notice it read as a clean green job while
something downstream waited on a merge that was never coming.
@sydorovdmytro
sydorovdmytro requested a review from a team as a code owner September 1, 2026 08:59
… suite hold

Review caught two things. A commit status was inheriting the check-suite
watermark: a running e2e rerun would vouch for an unrelated Netlify failure and
hold it for the full wait budget. Statuses carry no suite id and have no rerun
concept, so they now bail on their own, before the check-run hold is considered.

The comment and test name also claimed a broken PR always bails in seconds. It
does not: the watermark is not scoped to the failing check's name, so an
unrelated newer suite holds the bail to max_attempts. Name-scoping was tried and
reverted because it breaks the cancelled path it borrows from - a replacement
queued behind an earlier job has not published a check-run to match on. The
imprecision is now documented rather than overstated; it stays fail-closed.

@loft-bot loft-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Panel review: 0 blocking, 4 quality notes, 9 lanes. The merge-safety invariant holds — cr_newer_suite_pending > 0 implies pending > 0, so a held failure can never reach the green gate, and the widened unresolved watermark is what keeps a same-suite failure bailing on the first poll.

What was checked

Checked: correctness, security, test-quality, reuse, dead-code, typos, coverage gaps (analyzer + independent verifier), pr-metadata, cross-PR symbol conflicts. Skipped: e2e ×4, infra, operability, arch, entry-point-fidelity, layout (no e2e or infra files; no Go control-plane loop or migration; no structural, interface or dependency change; no customer-bug-plus-e2e pair; no renames). All three coverage-gap candidates were refuted on verification — notably, new test #6 already ships the both-failing fixture, and the timed_out/action_required variants traverse byte-identical lines.

PR-level notes

  • nit — Test plan says "132 pass, 0 fail (5 new)"; the diff adds 6 new @test blocks.
  • consider — The body closes DEVOPS-1454 (correct), but every new test and the new section header is tagged devops-1452 — a different, already-shipped issue (resumable cut-release, #246). Retag so a later grep devops-1454 finds this suite.

Quality notes (non-blocking)

  • src/wait-for-ci.sh:384consider — when a commit status and a check-run both fail on the same poll, the st_failed bail exits first and reports only the status context, so the failing check-run name is no longer logged. (The counts line still shows failed=1, so what is lost is which check, not the fact.) The deleted combined bail named both. Merging the condition — [ "$st_failed" -gt 0 ] || { [ "$cr_real_failed" -gt 0 ] && [ "$cr_newer_suite_pending" -eq 0 ]; } — with the old awk 'NF' | paste -sd, - detail restores it without weakening the deliberate "statuses are never held" rule.
  • src/wait-for-ci.sh (file-level; timeout line ~454) — consider — the new held-failure path can now exhaust max_attempts, and the timeout annotation then reads Last error: none (checks were still pending), naming neither the held failure nor the rerun being waited on, even though cr_real_failed_detail was printed on every intermediate poll. The code comment concedes this for the cross-check case, but disclosure is not a fix and the fix is local to that one line.
  • test/wait-for-ci.bats:1081 (and :1098) — consider — both commit-status tests assert only the substring A commit status reported failure, never the ::error:: prefix, while the check-run path gets a dedicated notice-vs-error test. A revert of just this branch to ::notice:: keeps both green. This becomes blocking if the commit-status refusal regresses to ::notice:: — the "green job that did nothing" mode that made all three cited incidents slow to diagnose.
  • README.md:108consider — "The job keeps its continue-on-error safety net, so it cannot turn a caller's CI red" is asserted here and twice more in the script comments, but sibling open PR #239 changes the reusable workflow to continue-on-error: ${{ !inputs.auto-merge }} and rewrites this same README region. Whichever merges second, this rationale is stale for auto-merge: true callers — the conclusion still holds (an ::error:: annotation cannot fail a job), only the stated reason does not. Worth coordinating with #239, which also conflicts textually here.
@sydorovdmytro
sydorovdmytro merged commit 819d89a into main Sep 1, 2026
9 checks passed
@sydorovdmytro
sydorovdmytro deleted the devops-1454/auto-approve-failure-hold branch September 1, 2026 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants