feat(run-ginkgo): add a validated flake-attempts input - #247
feat(run-ginkgo): add a validated flake-attempts input#247sydorovdmytro wants to merge 8 commits into
Conversation
Ginkgo retries only the spec that failed, so a flake budget costs one spec's runtime rather than a re-run of the whole job. Callers that must not be reddened by an infra flake (a release bump PR gating a cut) can set 2; everything else stays at ginkgo's default of 1 so the signal stays honest. Rejects a non-integer instead of passing it through: ginkgo would reject it too, but only after suite setup, turning a typo into a failed job minutes later.
Ginkgo reports a flaked spec as State "passed", so the summary counted it as a clean pass. Turning on flake-attempts would then erase the evidence that a suite is degrading - on a release bump PR, exactly the signal worth keeping. Adds a flaked line using the same predicate ginkgo uses for CountOfFlakedSpecs: passed, MaxFlakeAttempts > 1, NumAttempts > 1.
…ed on a retry Printing an unbroken "All tests passed!" above a flaked count contradicts it. Saying so in the headline is the point of tracking flakes at all.
The fixture's three specs all pass, so the count is 3/3, not 2/3.
loft-bot
left a comment
There was a problem hiding this comment.
Panel review: 1 blocking, 4 quality notes, 7 lanes.
Blocking concerns
.github/actions/run-ginkgo/test/execute-tests.bats:187— the two negative-onlyflake-attemptscases stay green even when the script never invokes ginkgo on the default path.
What was checked
Checked: correctness, security, test-quality, operability, architecture, typos, PR metadata, cross-PR symbol conflicts. Skipped: the e2e-family and coverage-gap lanes (this repo has no e2e*/ suite), infra (no workflow, Helm, Terraform, Dockerfile or k8s manifest changed — only a composite action manifest), reuse (no new helper; the added jq predicate follows the shape of its four sibling lines), dead-code (nothing deleted or renamed), layout coherence (no renames).
Verified rather than assumed: ginkgo v2.32.1 really does emit NumAttempts and MaxFlakeAttempts in its JSON report, so the new hand-written fixture is faithful and the flaked predicate does fire in production, matching upstream CountOfFlakedSpecs exactly. The blocking finding was reproduced by running the real bats suite against a mutated script.
PR-level notes
- nit — the issue is linked as "Part of DEVOPS-1453."; the repo convention is
Closes <ID>, orReferences <ID>for a sibling PR in a multi-repo change, which is the case that fits here. - nit — the test plan says "89 pass, 0 fail (5 new)"; the tree has 90 bats cases with 7 new. The 5 matches only
execute-tests.bats, omitting the 2 added later for the flaked count — the plan looks like it was not refreshed as the PR grew.
Quality notes (non-blocking)
- consider
.github/actions/run-ginkgo/action.yml:46— this 226-char description widens the generated DESCRIPTION column from 141 to 253 chars across all 12 rows, since auto-doc pads every cell to the widest. Convention here is a short description with the rationale as README prose;rerun-failed-onlyhas its own section butflake-attemptsgets none, so the when-to-use-2 guidance and the new🔁 Flaked (passed on retry)line have nowhere durable to live. - consider
.github/actions/run-ginkgo/test/generate-summary.bats:288— "…even with retries enabled" runs againstall-passed.json, which carries no flake fields, so retries are never actually exercised; dropping theNumAttempts > 1clause entirely leaves it green. The fixture needed is already present as the "clean pass" entry inwith-flakes.json(MaxFlakeAttempts: 2, NumAttempts: 1). This becomes blocking if the sibling test's exact-count assertion is ever relaxed, since that clause would then be wholly untested. - consider
.github/actions/run-ginkgo/src/execute-tests.sh:49—(( … )) && GINKGO_ARGS+=(…)departs from theif … thenform used by the three sibling conditional appends in this same file. Harmless today because statements follow it, but it exits non-zero underset -ethe moment it becomes the last statement of a block or moves into a function. - nit
.github/actions/run-ginkgo/src/execute-tests.sh:48— a leading-zero value such as08passes the^[0-9]+$regex but is an invalid octal literal in(( )), so bash prints a rawvalue too great for baseerror alongside the intended warning. Worth noting a bare10#fix is not a drop-in: it would accept08and then forward08to ginkgo, which rejects it (Go'sflagpackage parses ints with base auto-detection), so the forwarded value needs normalizing too.
…ive one A negative-only assertion greps a file the ginkgo mock may never have written, so it passes just as well when the script stopped doing anything: "the flag was correctly omitted" and "the suite never ran" look identical. Verified by mutation - an early exit before invoking ginkgo left every case green, including the one that owns the default path, which is the property this input rests on. The clean-run summary test ran against a fixture with no flake fields at all, so the NumAttempts > 1 clause was never exercised and could be deleted with the test still green. It now uses a fixture with retries enabled and none used. Also normalize leading zeros before the arithmetic: "08" satisfies ^[0-9]+$ but is an invalid octal literal, so (( )) printed its own error next to the warning, and forwarding it verbatim would fail in ginkgo too. Move the when-to-use guidance out of the input description and into README prose, where rerun-failed-only's already lives - auto-doc pads every cell to the widest, so a 226-char description widened the whole generated table.
…acuously Not part of this PR's change, but the same defect the review blocked on, three lines away in the same file. Mutating the script to never invoke ginkgo left these three green: they grep an argv file the mock never wrote, so `! grep` succeeds on a missing file and 'the flag was correctly omitted' reads the same as 'the suite never ran'. With the positive half added, that mutation now leaves only 'creates test-reports directory' green, which is correct - it asserts a side effect that happens before ginkgo is invoked.
Bash wraps at signed 64-bit while ginkgo's Go int flag rejects out-of-range values, so the comparison was being decided by the overflow. 18446744073709551617 wraps to 1: no warning, no flag, and the retries the caller asked for silently did not happen. 18446744073709551618 wraps to 2: it passed the >1 test and forwarded the original huge string for ginkgo to reject after the whole suite had been set up. The length bound now runs first, the same guard and the same reasoning as numeric_or_default in auto-approve-bot-prs. Also pairs the last same-class withhold test in rerun-integration.bats with a positive assertion. With that, mutating execute-tests.sh to never invoke ginkgo leaves green only tests that genuinely do not depend on it: one pre-ginkgo side effect, and three that exercise generate-summary.sh instead.
…scription Trimming it to a bare README pointer was over-correction. The width argument does not force it: the generated table is sized by upload-report's 138-char cell, and flake-attempts now renders at 132, so the widest row is 215 - identical to main. A caller reading the description in editor tooling can otherwise reasonably set this on a normal PR, which is exactly the misuse the guidance prevents.
References DEVOPS-1453
One flaky spec out of 190 reddened the
v0.34.8-rc.2bump PR and stalled thatrelease cut for two hours.
run-ginkgohad no flake-retry knob, so the onlyoptions were a manual re-run or nothing.
Adds
flake-attempts. Ginkgo retries only the spec that failed, so the cost isone spec's runtime, not a whole job. Default stays
1(ginkgo's own default),which adds no flag at all, so this is inert for every existing caller — the
release-branch wiring that actually sets
2is a separate vcluster-pro PR, whichis why this references rather than closes the issue.
Retried specs are not hidden. Ginkgo reports a spec that passed on a retry as
State: passed, so the summary counts them separately using ginkgo's ownCountOfFlakedSpecspredicate, and qualifies the headline rather than printing anunbroken "All tests passed!" above a flaked count. Without that, turning retries
on would erase the evidence that a suite is degrading — on a release bump PR,
exactly the signal worth keeping.
A value that is not an integer >= 1 warns and is dropped rather than forwarded:
ginkgo rejects it too, but only after suite setup, which turns a typo into a
failed job minutes later. Leading zeros are normalized, since
08satisfies^[0-9]+$but is an invalid octal literal in(( ))and is also rejected by Go'sflag package.
Test plan
make test-run-ginkgo— 97 pass, 0 fail (96 bats cases; 15 new)shellcheckon both changed scripts,make check-docs— cleanexit 0beforeinvoking ginkgo now fails the default-path test (it left all cases green
before), and deleting the
NumAttempts > 1clause now fails the clean-run testState: passedwithNumAttempts: 2, and that the summary reports it