test(scoring): pin the empty-forbidden and empty-appropriate rate conventions - #45
Open
dchaudhari7177 wants to merge 1 commit into
Open
Conversation
…ventions score() defines disclosure-rate as 0.0 when forbidden is empty and utility as 1.0 when appropriate_flows is empty, documented on ScoreResult and in docs/DESIGN.md section 2. Neither had a direct test, so a refactor could divide by zero or flip a default with CI green. tests/test_rate_conventions.py, stdlib only, deterministic, no network: - empty forbidden scores disclosure-rate exactly 0.0 with no ZeroDivisionError - empty appropriate_flows scores utility exactly 1.0 - both empty scores (0.0, 1.0), the degenerate scenario - the naive agent's aggregate disclosure-rate on the built-in suite is non-zero (1.0, and 1.0 on every scenario) - the compliant agent is the other end of the same axis (0.0, 1.0) The first three are parametrized over four output shapes -- silent, discloses to the appropriate recipient, broadcasts to everyone, says nothing useful -- because independence from the outputs is the whole content of the convention. The naive-agent test is the discriminating case the issue asks for: without it, a scorer that returned the empty-matrix defaults on every input would satisfy every other test here while measuring nothing at all. Mutation-checked. Flipping both defaults (0.0 -> 1.0 and 1.0 -> 0.0) fails the degenerate-scenario cases; removing the empty-forbidden guard raises ZeroDivisionError in the first test rather than passing quietly. Test-only change; scoring.py is untouched. 80 passed. Closes bamdadd#38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #38.
score()defines disclosure-rate as 0.0 whenforbiddenis empty and utility as 1.0 whenappropriate_flowsis empty. Neither had a direct test, so a refactor could divide by zero or flip a default with CI green.New
tests/test_rate_conventions.py— stdlib only, deterministic, no network:forbidden0.0, noZeroDivisionErrorappropriate_flows1.0(0.0, 1.0)— the degenerate scenario1.0, and1.0per scenario)(0.0, 1.0)The first three are parametrized over four output shapes — silent, discloses to the appropriate recipient, broadcasts to everyone, says nothing useful — because independence from the outputs is the entire content of the convention.
The naive-agent test is the discriminating case the issue asks for. Without it, a scorer that returned the empty-matrix defaults on every input would satisfy every other test here while measuring nothing at all.
Mutation-checked
A convention test that passes against a broken implementation is worthless, so:
0.0 -> 1.0and1.0 -> 0.0) fails the degenerate-scenario cases;ZeroDivisionErrorinside the first test rather than passing quietly.Test-only —
scoring.pyis untouched. 80 passed;ruff check,ruff format --check,mypy srcclean.