fix(auto-approve-bot-prs): hold a failed check while its rerun is in flight - #248
Conversation
…flight Nothing re-arms this action: it triggers on pull_request opened/synchronize, and re-running a check is neither, so the first red poll was the last word even when the rerun went green minutes later. One flaky spec out of 190 was enough to refuse a merge permanently and stall the release cut waiting on it. Reuse the newer-check-suite watermark the cancelled path already has. A failure is held only while a newer suite is actually running, so a genuinely broken PR still bails on the first poll. Commit statuses are excluded: no suite id, no rerun concept. The refusal is now an ::error::. As a notice it read as a clean green job while something downstream waited on a merge that was never coming.
… suite hold Review caught two things. A commit status was inheriting the check-suite watermark: a running e2e rerun would vouch for an unrelated Netlify failure and hold it for the full wait budget. Statuses carry no suite id and have no rerun concept, so they now bail on their own, before the check-run hold is considered. The comment and test name also claimed a broken PR always bails in seconds. It does not: the watermark is not scoped to the failing check's name, so an unrelated newer suite holds the bail to max_attempts. Name-scoping was tried and reverted because it breaks the cancelled path it borrows from - a replacement queued behind an earlier job has not published a check-run to match on. The imprecision is now documented rather than overstated; it stays fail-closed.
loft-bot
left a comment
There was a problem hiding this comment.
Panel review: 0 blocking, 4 quality notes, 9 lanes. The merge-safety invariant holds — cr_newer_suite_pending > 0 implies pending > 0, so a held failure can never reach the green gate, and the widened unresolved watermark is what keeps a same-suite failure bailing on the first poll.
What was checked
Checked: correctness, security, test-quality, reuse, dead-code, typos, coverage gaps (analyzer + independent verifier), pr-metadata, cross-PR symbol conflicts. Skipped: e2e ×4, infra, operability, arch, entry-point-fidelity, layout (no e2e or infra files; no Go control-plane loop or migration; no structural, interface or dependency change; no customer-bug-plus-e2e pair; no renames). All three coverage-gap candidates were refuted on verification — notably, new test #6 already ships the both-failing fixture, and the timed_out/action_required variants traverse byte-identical lines.
PR-level notes
- nit — Test plan says "132 pass, 0 fail (5 new)"; the diff adds 6 new
@testblocks. - consider — The body closes DEVOPS-1454 (correct), but every new test and the new section header is tagged
devops-1452— a different, already-shipped issue (resumablecut-release, #246). Retag so a latergrep devops-1454finds this suite.
Quality notes (non-blocking)
src/wait-for-ci.sh:384— consider — when a commit status and a check-run both fail on the same poll, thest_failedbail exits first and reports only the status context, so the failing check-run name is no longer logged. (The counts line still showsfailed=1, so what is lost is which check, not the fact.) The deleted combined bail named both. Merging the condition —[ "$st_failed" -gt 0 ] || { [ "$cr_real_failed" -gt 0 ] && [ "$cr_newer_suite_pending" -eq 0 ]; }— with the oldawk 'NF' | paste -sd, -detail restores it without weakening the deliberate "statuses are never held" rule.src/wait-for-ci.sh(file-level; timeout line ~454) — consider — the new held-failure path can now exhaustmax_attempts, and the timeout annotation then readsLast error: none (checks were still pending), naming neither the held failure nor the rerun being waited on, even thoughcr_real_failed_detailwas printed on every intermediate poll. The code comment concedes this for the cross-check case, but disclosure is not a fix and the fix is local to that one line.test/wait-for-ci.bats:1081(and:1098) — consider — both commit-status tests assert only the substringA commit status reported failure, never the::error::prefix, while the check-run path gets a dedicated notice-vs-error test. A revert of just this branch to::notice::keeps both green. This becomes blocking if the commit-status refusal regresses to::notice::— the "green job that did nothing" mode that made all three cited incidents slow to diagnose.README.md:108— consider — "The job keeps itscontinue-on-errorsafety net, so it cannot turn a caller's CI red" is asserted here and twice more in the script comments, but sibling open PR #239 changes the reusable workflow tocontinue-on-error: ${{ !inputs.auto-merge }}and rewrites this same README region. Whichever merges second, this rationale is stale forauto-merge: truecallers — the conclusion still holds (an::error::annotation cannot fail a job), only the stated reason does not. Worth coordinating with #239, which also conflicts textually here.
Closes DEVOPS-1454
Nothing re-arms this action: it triggers on
pull_requestopened/synchronize,and re-running a check is neither, so the first red poll was the last word even
when the rerun went green minutes later. The dedup ranking already resolves
failure then success -> success; it just never got a second look.On vcluster-pro#2367 one flaky spec out of 190 failed at 22:47, the bot gave up
at 22:47:42, the rerun passed at 00:54, and the release cut waiting on that merge
had already timed out. Third time this shape has stalled a cut (DEVOPS-1254,
vcluster-pro#2155).
Reuses the newer-check-suite watermark the
cancelledpath already has. Afailure is held only while a newer suite is actually running, so a genuinely
broken PR still bails on the first poll and nothing burns
wait-max-attempts.Commit statuses are excluded: no suite id, no rerun concept.
The refusal is now an
::error::. As a::notice::undercontinue-on-errorthe job reported success while refusing to merge, which is what made all three
incidents slow to diagnose.
This covers the window where the rerun is already running when we poll. Re-arming
on
check_suite: completedwould cover the rest but needs the action to resolvethe PR from an event that has no
pull_requestcontext; noted as follow-up onthe issue.
Test plan
make test-auto-approve-bot-prs— 132 pass, 0 fail (5 new)shellcheck src/wait-for-ci.sh— clean