Skip to content

fix: skip Putnam samples graded in prior runs - #12

Open
sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/putnam-grade-reruns
Open

fix: skip Putnam samples graded in prior runs#12
sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/putnam-grade-reruns

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Fixes #11.

grade_gemini_sample() currently constructs the expected grade filename using the timestamp of the current grading run and only skips when that exact path already exists. Because the timestamp changes on every invocation, samples graded during an earlier run are sent to the grader again.

This change:

  • detects an existing grade for the same grader model and sample stem regardless of the previous run's timestamp;
  • matches the exact grader model and expected YYYYMMDD-HHMMSS timestamp format, so a differently named grader with a shared prefix is not treated as the same grader;
  • preserves the existing timestamped filename format for new grades;
  • adds regression coverage for both prior-run skipping and distinct grader models.

Validation

A focused unittest regression test is included for the reported failure mode, plus a case ensuring a differently named grader with a shared prefix does not cause an incorrect skip.

The tests were not run locally in this environment; no local test result is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant