Skip to content

fix: report the weight plan recorded, not a second one - #67

Merged
deepskandpal merged 1 commit into
deepskandpal:mainfrom
dchaudhari7177:fix/summarise-weight
Aug 30, 2026
Merged

fix: report the weight plan recorded, not a second one#67
deepskandpal merged 1 commit into
deepskandpal:mainfrom
dchaudhari7177:fix/summarise-weight

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #62

Your worked example, run against this branch:

by_stratum      : {'fail': 15, 'pass': 25}
carried on rows : {'fail': 1.0, 'pass': 3.0}
reported weights: {'fail': 1.0, 'pass': 3.0}   ← agrees

old formula would report: {'fail': 6.0, 'pass': 3.6}

The fix

Rather than correcting the formula in summarise to match plan's, I removed the second derivation entirely: the weight is read off the Selection rows.

weights[selection.stratum] = round(selection.weight, 3)

Recomputing it correctly would leave the same underlying situation you describe — two places deriving the same quantity, and a later edit to one silently reintroducing the disagreement. Now plan holds the only definition, and summarise reports what the rows carry, so the two cannot drift by construction.

It also happens to be less code, and it is safe: every Selection in a stratum is assigned the same weight in the same loop iteration, so reading it per-row is unambiguous.

Scope

This is only the narrower point — that the two disagreed. I have deliberately not touched which weight is right, since #60 establishes that neither is a valid inclusion weight given the deterministic within-stratum selection. Whatever #60 concludes will change plan, and summarise will follow it automatically now instead of needing a matching edit.

Tests

Seven new cases in tests/test_sampling.py:

  • test_the_reported_weight_is_the_weight_the_rows_carry — the invariant, stated directly
  • test_the_reported_weight_is_stratum_size_over_the_number_taken — your 90/15/40 example with both figures pinned
  • test_a_weight_is_never_the_suite_size_over_a_stratum_count — computes the old formula and asserts the report differs from it, so reintroducing it as a simplification fails loudly
  • test_a_fully_sampled_stratum_weighs_one — taking every example in a stratum owes no correction
  • test_weights_reconstruct_the_stratum_sizes — the property that makes a weight usable at all: count × weight is the stratum size. The old formula fails this
  • test_a_stratum_that_contributed_nothing_is_absent_from_the_weights
  • test_summarising_an_empty_plan_reports_no_weights

Reverting only sampling.py while keeping the tests fails 5 of the 7.

Verification

pytest — 197 passed. The one failure, test_docs_in_sync.py::test_verify_script_is_executable, reproduces identically on pristine main (it asserts a POSIX executable bit; I am on Windows).

ruff check and ruff format --check clean.

summarise computed len(rows) / count -- the whole suite over one
stratum's count -- putting a suite-level numerator over a stratum-level
denominator, which is not a quantity that means anything. On a
90-example suite with 15 judge-fails and a budget of 40 it reported 6.0
and 3.6 where the Selection rows carried 1.0 and 3.0.

Read the weight off the selections instead of deriving a second one.
Nothing consumed the reported figure, so no published number was wrong;
the cost was that one module offered two candidate weights with nothing
to say which was intended. There is now exactly one definition, in
plan().
@deepskandpal

Copy link
Copy Markdown
Owner

Reviewed. Correct diagnosis of a real bug, and the comment explaining it is better than the fix needed to be.

summarise() computed len(rows) / count, a suite-level numerator over a stratum-level denominator, while plan() recorded available[name] / take. Two different definitions of the same word in one module, with nothing to say which was intended. Your worked example showing 6.0 and 3.6 reported against 1.0 and 3.0 carried is the clearest possible way to present it.

You are also right that nothing consumed the reported figure, so no published number was wrong. Saying so rather than overstating the severity is appreciated.

One piece of context you could not have had. Selection.weight itself is under review in #60, because plan() selects the lowest-confidence rows deterministically inside each stratum, which means inclusion is 0 or 1 and the recorded weight is not a valid inclusion weight at all. That does not affect this PR: making one module agree with itself is right regardless of what the number turns out to mean, and this makes #60 easier rather than harder by leaving exactly one definition to reason about.

CI has not run yet. The workflow is held pending approval, so guard is the only check reporting.

@deepskandpal
deepskandpal merged commit 5251b08 into deepskandpal:main Aug 30, 2026
3 checks passed
deepskandpal added a commit that referenced this pull request Aug 30, 2026
#57 and #67 landed on origin while #28, #31 and #54 were being merged locally.
tests/test_flow.py conflicted: #28 added a per-criterion attribution test and
#58 added four ambiguity-disclosure tests, and git tangled them because both
open with identical fixture setup. Resolved as the union of both, reconstructed
from each side's full version rather than hand-patched, so no assertion was
silently dropped in the middle of a hunk.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants