Skip to content

feat(run-ginkgo): add a validated flake-attempts input - #247

Open
sydorovdmytro wants to merge 8 commits into
mainfrom
devops-1453/e2e-flake-attempts
Open

feat(run-ginkgo): add a validated flake-attempts input#247
sydorovdmytro wants to merge 8 commits into
mainfrom
devops-1453/e2e-flake-attempts

Conversation

@sydorovdmytro

@sydorovdmytro sydorovdmytro commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

References DEVOPS-1453

One flaky spec out of 190 reddened the v0.34.8-rc.2 bump PR and stalled that
release cut for two hours. run-ginkgo had no flake-retry knob, so the only
options were a manual re-run or nothing.

Adds flake-attempts. Ginkgo retries only the spec that failed, so the cost is
one spec's runtime, not a whole job. Default stays 1 (ginkgo's own default),
which adds no flag at all, so this is inert for every existing caller — the
release-branch wiring that actually sets 2 is a separate vcluster-pro PR, which
is why this references rather than closes the issue.

Retried specs are not hidden. Ginkgo reports a spec that passed on a retry as
State: passed, so the summary counts them separately using ginkgo's own
CountOfFlakedSpecs predicate, and qualifies the headline rather than printing an
unbroken "All tests passed!" above a flaked count. Without that, turning retries
on would erase the evidence that a suite is degrading — on a release bump PR,
exactly the signal worth keeping.

A value that is not an integer >= 1 warns and is dropped rather than forwarded:
ginkgo rejects it too, but only after suite setup, which turns a typo into a
failed job minutes later. Leading zeros are normalized, since 08 satisfies
^[0-9]+$ but is an invalid octal literal in (( )) and is also rejected by Go's
flag package.

Test plan

  • make test-run-ginkgo — 97 pass, 0 fail (96 bats cases; 15 new)
  • shellcheck on both changed scripts, make check-docs — clean
  • Mutation-tested both directions rather than asserted: an early exit 0 before
    invoking ginkgo now fails the default-path test (it left all cases green
    before), and deleting the NumAttempts > 1 clause now fails the clean-run test
  • Verified against a real ginkgo report that a retried-then-passed spec is
    State: passed with NumAttempts: 2, and that the summary reports it
Ginkgo retries only the spec that failed, so a flake budget costs one spec's
runtime rather than a re-run of the whole job. Callers that must not be reddened
by an infra flake (a release bump PR gating a cut) can set 2; everything else
stays at ginkgo's default of 1 so the signal stays honest.

Rejects a non-integer instead of passing it through: ginkgo would reject it too,
but only after suite setup, turning a typo into a failed job minutes later.
@sydorovdmytro
sydorovdmytro requested a review from a team as a code owner September 1, 2026 08:51
Ginkgo reports a flaked spec as State "passed", so the summary counted it as a
clean pass. Turning on flake-attempts would then erase the evidence that a suite
is degrading - on a release bump PR, exactly the signal worth keeping.

Adds a flaked line using the same predicate ginkgo uses for CountOfFlakedSpecs:
passed, MaxFlakeAttempts > 1, NumAttempts > 1.
…ed on a retry

Printing an unbroken "All tests passed!" above a flaked count contradicts it.
Saying so in the headline is the point of tracking flakes at all.
The fixture's three specs all pass, so the count is 3/3, not 2/3.

@loft-bot loft-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Panel review: 1 blocking, 4 quality notes, 7 lanes.

Blocking concerns

  • .github/actions/run-ginkgo/test/execute-tests.bats:187 — the two negative-only flake-attempts cases stay green even when the script never invokes ginkgo on the default path.

What was checked

Checked: correctness, security, test-quality, operability, architecture, typos, PR metadata, cross-PR symbol conflicts. Skipped: the e2e-family and coverage-gap lanes (this repo has no e2e*/ suite), infra (no workflow, Helm, Terraform, Dockerfile or k8s manifest changed — only a composite action manifest), reuse (no new helper; the added jq predicate follows the shape of its four sibling lines), dead-code (nothing deleted or renamed), layout coherence (no renames).

Verified rather than assumed: ginkgo v2.32.1 really does emit NumAttempts and MaxFlakeAttempts in its JSON report, so the new hand-written fixture is faithful and the flaked predicate does fire in production, matching upstream CountOfFlakedSpecs exactly. The blocking finding was reproduced by running the real bats suite against a mutated script.

PR-level notes

  • nit — the issue is linked as "Part of DEVOPS-1453."; the repo convention is Closes <ID>, or References <ID> for a sibling PR in a multi-repo change, which is the case that fits here.
  • nit — the test plan says "89 pass, 0 fail (5 new)"; the tree has 90 bats cases with 7 new. The 5 matches only execute-tests.bats, omitting the 2 added later for the flaked count — the plan looks like it was not refreshed as the PR grew.

Quality notes (non-blocking)

  • consider .github/actions/run-ginkgo/action.yml:46 — this 226-char description widens the generated DESCRIPTION column from 141 to 253 chars across all 12 rows, since auto-doc pads every cell to the widest. Convention here is a short description with the rationale as README prose; rerun-failed-only has its own section but flake-attempts gets none, so the when-to-use-2 guidance and the new 🔁 Flaked (passed on retry) line have nowhere durable to live.
  • consider .github/actions/run-ginkgo/test/generate-summary.bats:288 — "…even with retries enabled" runs against all-passed.json, which carries no flake fields, so retries are never actually exercised; dropping the NumAttempts > 1 clause entirely leaves it green. The fixture needed is already present as the "clean pass" entry in with-flakes.json (MaxFlakeAttempts: 2, NumAttempts: 1). This becomes blocking if the sibling test's exact-count assertion is ever relaxed, since that clause would then be wholly untested.
  • consider .github/actions/run-ginkgo/src/execute-tests.sh:49(( … )) && GINKGO_ARGS+=(…) departs from the if … then form used by the three sibling conditional appends in this same file. Harmless today because statements follow it, but it exits non-zero under set -e the moment it becomes the last statement of a block or moves into a function.
  • nit .github/actions/run-ginkgo/src/execute-tests.sh:48 — a leading-zero value such as 08 passes the ^[0-9]+$ regex but is an invalid octal literal in (( )), so bash prints a raw value too great for base error alongside the intended warning. Worth noting a bare 10# fix is not a drop-in: it would accept 08 and then forward 08 to ginkgo, which rejects it (Go's flag package parses ints with base auto-detection), so the forwarded value needs normalizing too.
Comment thread .github/actions/run-ginkgo/test/execute-tests.bats
…ive one

A negative-only assertion greps a file the ginkgo mock may never have written,
so it passes just as well when the script stopped doing anything: "the flag was
correctly omitted" and "the suite never ran" look identical. Verified by
mutation - an early exit before invoking ginkgo left every case green, including
the one that owns the default path, which is the property this input rests on.

The clean-run summary test ran against a fixture with no flake fields at all, so
the NumAttempts > 1 clause was never exercised and could be deleted with the test
still green. It now uses a fixture with retries enabled and none used.

Also normalize leading zeros before the arithmetic: "08" satisfies ^[0-9]+$ but
is an invalid octal literal, so (( )) printed its own error next to the warning,
and forwarding it verbatim would fail in ginkgo too.

Move the when-to-use guidance out of the input description and into README prose,
where rerun-failed-only's already lives - auto-doc pads every cell to the widest,
so a 226-char description widened the whole generated table.
…acuously

Not part of this PR's change, but the same defect the review blocked on, three
lines away in the same file. Mutating the script to never invoke ginkgo left
these three green: they grep an argv file the mock never wrote, so `! grep`
succeeds on a missing file and 'the flag was correctly omitted' reads the same
as 'the suite never ran'.

With the positive half added, that mutation now leaves only 'creates
test-reports directory' green, which is correct - it asserts a side effect that
happens before ginkgo is invoked.
Bash wraps at signed 64-bit while ginkgo's Go int flag rejects out-of-range
values, so the comparison was being decided by the overflow.
18446744073709551617 wraps to 1: no warning, no flag, and the retries the caller
asked for silently did not happen. 18446744073709551618 wraps to 2: it passed the
>1 test and forwarded the original huge string for ginkgo to reject after the
whole suite had been set up.

The length bound now runs first, the same guard and the same reasoning as
numeric_or_default in auto-approve-bot-prs.

Also pairs the last same-class withhold test in rerun-integration.bats with a
positive assertion. With that, mutating execute-tests.sh to never invoke ginkgo
leaves green only tests that genuinely do not depend on it: one pre-ginkgo side
effect, and three that exercise generate-summary.sh instead.
…scription

Trimming it to a bare README pointer was over-correction. The width argument does
not force it: the generated table is sized by upload-report's 138-char cell, and
flake-attempts now renders at 132, so the widest row is 215 - identical to main.

A caller reading the description in editor tooling can otherwise reasonably set
this on a normal PR, which is exactly the misuse the guidance prevents.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants