fix: report the weight plan recorded, not a second one - #67
Conversation
summarise computed len(rows) / count -- the whole suite over one stratum's count -- putting a suite-level numerator over a stratum-level denominator, which is not a quantity that means anything. On a 90-example suite with 15 judge-fails and a budget of 40 it reported 6.0 and 3.6 where the Selection rows carried 1.0 and 3.0. Read the weight off the selections instead of deriving a second one. Nothing consumed the reported figure, so no published number was wrong; the cost was that one module offered two candidate weights with nothing to say which was intended. There is now exactly one definition, in plan().
|
Reviewed. Correct diagnosis of a real bug, and the comment explaining it is better than the fix needed to be.
You are also right that nothing consumed the reported figure, so no published number was wrong. Saying so rather than overstating the severity is appreciated. One piece of context you could not have had. CI has not run yet. The workflow is held pending approval, so |
#57 and #67 landed on origin while #28, #31 and #54 were being merged locally. tests/test_flow.py conflicted: #28 added a per-criterion attribution test and #58 added four ambiguity-disclosure tests, and git tangled them because both open with identical fixture setup. Resolved as the union of both, reconstructed from each side's full version rather than hand-patched, so no assertion was silently dropped in the middle of a hunk.
Closes #62
Your worked example, run against this branch:
The fix
Rather than correcting the formula in
summariseto matchplan's, I removed the second derivation entirely: the weight is read off theSelectionrows.Recomputing it correctly would leave the same underlying situation you describe — two places deriving the same quantity, and a later edit to one silently reintroducing the disagreement. Now
planholds the only definition, andsummarisereports what the rows carry, so the two cannot drift by construction.It also happens to be less code, and it is safe: every
Selectionin a stratum is assigned the sameweightin the same loop iteration, so reading it per-row is unambiguous.Scope
This is only the narrower point — that the two disagreed. I have deliberately not touched which weight is right, since #60 establishes that neither is a valid inclusion weight given the deterministic within-stratum selection. Whatever #60 concludes will change
plan, andsummarisewill follow it automatically now instead of needing a matching edit.Tests
Seven new cases in
tests/test_sampling.py:test_the_reported_weight_is_the_weight_the_rows_carry— the invariant, stated directlytest_the_reported_weight_is_stratum_size_over_the_number_taken— your 90/15/40 example with both figures pinnedtest_a_weight_is_never_the_suite_size_over_a_stratum_count— computes the old formula and asserts the report differs from it, so reintroducing it as a simplification fails loudlytest_a_fully_sampled_stratum_weighs_one— taking every example in a stratum owes no correctiontest_weights_reconstruct_the_stratum_sizes— the property that makes a weight usable at all:count × weightis the stratum size. The old formula fails thistest_a_stratum_that_contributed_nothing_is_absent_from_the_weightstest_summarising_an_empty_plan_reports_no_weightsReverting only
sampling.pywhile keeping the tests fails 5 of the 7.Verification
pytest— 197 passed. The one failure,test_docs_in_sync.py::test_verify_script_is_executable, reproduces identically on pristinemain(it asserts a POSIX executable bit; I am on Windows).ruff checkandruff format --checkclean.