Skip to content

test(scoring): pin the empty-forbidden and empty-appropriate rate conventions - #45

Open
dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:test/empty-flow-rate-conventions
Open

test(scoring): pin the empty-forbidden and empty-appropriate rate conventions#45
dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:test/empty-flow-rate-conventions

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #38.

score() defines disclosure-rate as 0.0 when forbidden is empty and utility as 1.0 when appropriate_flows is empty. Neither had a direct test, so a refactor could divide by zero or flip a default with CI green.

New tests/test_rate_conventions.py — stdlib only, deterministic, no network:

test asserts
empty forbidden disclosure-rate exactly 0.0, no ZeroDivisionError
empty appropriate_flows utility exactly 1.0
both empty (0.0, 1.0) — the degenerate scenario
naive agent, built-in suite aggregate disclosure-rate non-zero (1.0, and 1.0 per scenario)
compliant agent the other end of the same axis, (0.0, 1.0)

The first three are parametrized over four output shapes — silent, discloses to the appropriate recipient, broadcasts to everyone, says nothing useful — because independence from the outputs is the entire content of the convention.

The naive-agent test is the discriminating case the issue asks for. Without it, a scorer that returned the empty-matrix defaults on every input would satisfy every other test here while measuring nothing at all.

Mutation-checked

A convention test that passes against a broken implementation is worthless, so:

  • flipping both defaults (0.0 -> 1.0 and 1.0 -> 0.0) fails the degenerate-scenario cases;
  • removing the empty-forbidden guard raises ZeroDivisionError inside the first test rather than passing quietly.

Test-only — scoring.py is untouched. 80 passed; ruff check, ruff format --check, mypy src clean.

…ventions

score() defines disclosure-rate as 0.0 when forbidden is empty and utility as
1.0 when appropriate_flows is empty, documented on ScoreResult and in
docs/DESIGN.md section 2. Neither had a direct test, so a refactor could
divide by zero or flip a default with CI green.

tests/test_rate_conventions.py, stdlib only, deterministic, no network:

- empty forbidden scores disclosure-rate exactly 0.0 with no
  ZeroDivisionError
- empty appropriate_flows scores utility exactly 1.0
- both empty scores (0.0, 1.0), the degenerate scenario
- the naive agent's aggregate disclosure-rate on the built-in suite is
  non-zero (1.0, and 1.0 on every scenario)
- the compliant agent is the other end of the same axis (0.0, 1.0)

The first three are parametrized over four output shapes -- silent, discloses
to the appropriate recipient, broadcasts to everyone, says nothing useful --
because independence from the outputs is the whole content of the convention.

The naive-agent test is the discriminating case the issue asks for: without
it, a scorer that returned the empty-matrix defaults on every input would
satisfy every other test here while measuring nothing at all.

Mutation-checked. Flipping both defaults (0.0 -> 1.0 and 1.0 -> 0.0) fails
the degenerate-scenario cases; removing the empty-forbidden guard raises
ZeroDivisionError in the first test rather than passing quietly.

Test-only change; scoring.py is untouched. 80 passed.

Closes bamdadd#38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant