Add functional eval scenarios and CI reporting

[AshishKumar4]

This PR adds two more evals and a manual GitHub Actions workflow. The appointment eval checks booking limits, concurrent requests, duplicate IDs, cancellation, and persistence after the agent's changes are accepted. The expense eval uses two prompts in the same workspace to check cents-based splitting, balances, settlements, input errors, monthly totals, budgets, and whether the second code change preserves the first turn's data.

It also adds post-accept checks, so an eval can verify provisional Gadget code, accept the changes, reload the Gadget, and confirm that its behavior and stored data survive.

The workflow runs pnpm evals against local workerd, uploads results.json, and publishes one GitHub check. It accepts model and trial inputs, plus a report-only mode for collecting scores without requiring every task to pass.

---

Wall time: 1.78 seconds

Wall time: 0.45 seconds

[github-actions]

Preview: pr362-evals-scenarios-ci

https://pr362-evals-scenarios-ci-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

The capacity contract accepts any positive integer, but unique bookings are only exercised through capacity 3. The capacity-8 fixture below sends one repeated booking ID, so an implementation that enforces Math.min(slot.capacity, 3) passes every check while rejecting the fourth valid booking on larger slots. Please exercise more than three unique bookings against a larger capacity.

[ask-bonk]

This reload check only covers seq, so cancellation need not be durable. An implementation can remove cancel-1 from memory, persist cancel-3, and pass the pre-accept check; after reload cancel-1 reappears and the capacity-1 slot is oversold. Please verify the final cancel state after accepting as well.

[ask-bonk]

Turn two ends without accepting and reloading these changes. A Gadget that durably stores expenses but keeps every budget in an instance-local Map passes all budget assertions, even though the task requires Gadget-owned storage that survives restart. Add post-accept verification for the budget rows created in this turn.

[ask-bonk]

Equal balances do not prove that rejected inputs changed nothing. For example, the Gadget can insert the zero-amount or empty-split rows and return the expected errors; those rows contribute no balance, so this still passes while their IDs remain consumed. Retry the rejected IDs with valid data or otherwise observe that no rows were stored.

[ask-bonk]

Submitted 4 actionable inline review comments.

github run

[ask-bonk]

With reportOnly true this disables the action gate entirely, while the eval step above is still continue-on-error. In vitest-evals an unenforced report is advisory (shouldFail is false), even when the JSON records a non-eval/global-setup failure or zero eval cases. Therefore report-only runs can finish green for infrastructure failures, although the input promises only to tolerate failed trials / skip score enforcement. Please preserve a hard-failure path for invalid or infrastructure-failed reports while suppressing only qualitative eval misses.

[ask-bonk]

This counts successful responses and stored holders independently, so they need not describe the same bookings. For example, race-0..2 can return success while the implementation stores person-7..9; the accepted/error counts and three unique personIds all pass, but successful callers have no reservations. Please compare status.personIds with the people corresponding to the successful result indexes.

[ask-bonk]

status is captured before the cross-slot duplicate attempt, so this does not verify that the rejected call changed nothing in the original slot. A handler can update dupe-same from ana to ben, return DUPLICATE_BOOKING, and leave dupe-other empty; every assertion still passes. Re-read and validate the original slot after crossSlot.

[ask-bonk]

The turn-two categorized expenses are only observed through budgetStatus(). These assertions still pass if budget-1 and budget-2 are kept in a separate budget table and ignored by balances(), settlement(), and monthlyTotals(): the checked March balances remain exact and the global sum remains zero. That violates the unchanged LedgerApi contract for newly added expenses. Please assert the June total and the expected budget-ana/budget-ben balances (and settlement) after adding them.

[ask-bonk]

Submitted 4 actionable inline review comments.

github run

[ask-bonk]

• P2 appointment-desk.eval.ts:170: Reload verification does not attempt another booking. An in-memory capacity counter can reset and oversell after reload while slotStatus() still reads persisted rows. Require SLOT_FULL post-reload.
• P2 appointment-desk.eval.ts:119: UNKNOWN_SLOT may still consume "ghost". Define that slot afterward and retry the same booking ID to verify rejection changed nothing.
• P2 appointment-desk.eval.ts:103: Cancellation is only tested on a one-booking slot. Deleting every booking in the slot would pass. Cancel one booking from a multi-booking slot and verify the others remain.
• P2 appointment-desk.eval.ts:82: All successful bookings use distinct people. A DISTINCT personId status query incorrectly passes. Exercise two booking IDs for one person and verify personIds.length === bookedCount.
• P2 expense-ledger.eval.ts:251: Calendar validation only probes February 31. A validator that mishandles leap days or accepts April 31 passes. Add representative valid and invalid calendar boundaries.
• P2 expense-ledger.eval.ts:399: Categorized spending occurs only in June, so budgetStatus() may aggregate category spending across every month. Add the same category in another month and verify isolation.
• P2 expense-ledger.eval.ts:416: The post-upgrade duplicate omits category. Uniqueness incorrectly changed to (id, category) would pass; retry an existing ID with a different category.

github run

[ask-bonk]

• P2 expense-ledger.eval.ts:305: Reload checks only March/April data. All accepted May expenses can disappear after reload unnoticed. Verify May totals or balances too.
• P2 appointment-desk.eval.ts:99: Setup results are ignored here and at lines 131, 149-150. Valid operations can return errors while mutating state and still pass. Assert { ok: true }.
• P2 expense-ledger.eval.ts:201: Duplicate perPerson rows pass balance checks and inflate the settlement transfer bound. Assert unique person IDs.
• P2 expense-ledger.eval.ts:369: Only valid positive budgets are tested. Negative or fractional limits can be accepted despite the integer >= 0 contract.
• P2 expense-ledger.eval.ts:251: Date validation tests only an impossible, correctly formatted date. Malformed non-YYYY-MM-DD values remain untested.

github run

[ask-bonk]

• P2 packages/workshop-evals/evals/appointment-desk.eval.ts:76: startIso is never observable or verified. An implementation can discard appointment times entirely and still pass. Expose it through status/listing RPC and verify it survives reload.

github run

[ask-bonk]

• P2 appointment-desk.eval.ts:151: Duplicate IDs are only raced within one slot. Concurrent bookings using the same ID across different slots can both succeed. Race two slots and require one rejection.
• P2 appointment-desk.eval.ts:170: Reload verification never rechecks booking-ID uniqueness. Persisted bookings with an in-memory uniqueness index pass. Retry dupe-same after reload and require DUPLICATE_BOOKING.
• P2 expense-ledger.eval.ts:295: Each March payer has only one expense, allowing byPayer implementations that overwrite rather than sum. Add two same-month expenses from one payer.
• P2 expense-ledger.eval.ts:370: Budget replacement occurs only before spending. An implementation that resets accumulated spending when replacing a limit passes. Replace a budget after adding its expense.
• P2 expense-ledger.eval.ts:379: Each category has at most one new expense, so category spending may overwrite rather than aggregate. Add a second expense in one category.
• P2 expense-ledger.eval.ts:416: Turn two only rechecks duplicate-ID validation. The upgrade can drop INVALID_AMOUNT, EMPTY_SPLIT, and BAD_DATE handling while passing. Re-exercise those validations after the upgrade.

github run

[ask-bonk]

• P2 appointment-desk.eval.ts:103: Concurrent cancellation is untested. Racing cancellation of one booking could double-free capacity and permit overselling while all checks pass.
• P2 expense-ledger.eval.ts:113: Expense descriptions are never observable, so implementations may discard them. Add a listing RPC and verify descriptions persist across reload.

github run

[ask-bonk]

The turn-two settlement() call is the final state access, so the eval does not verify that this read leaves the ledger unchanged. An implementation can compute valid transfers from the balances captured above, then delete the expense rows (or persist settlement payments) before returning; every assertion here still passes even though the existing ledger has been destroyed. Please re-read balances/monthly totals after settlement() and require the pre-call state to remain intact.

[ask-bonk]

Submitted 1 actionable inline review comment.

github run

[ask-bonk]

• P2 appointment-desk.eval.ts:162: The empty-slot check only asserts bookedCount. Incorrect capacity or nonempty personIds still pass, violating slotStatus().

• P2 expense-ledger.eval.ts:249: No valid singleton split is exercised. An implementation rejecting every split with fewer than two participants passes despite singleton splits being valid.

• P2 expense-ledger.eval.ts:359: "uncategorized" is never budgeted. An implementation can treat uncategorized spending as permanently unbudgetable, despite setBudget() accepting any category.

github run

[ask-bonk]

P2 Duplicate slot IDs are only tested sequentially. A defineSlot() implementation that checks, awaits, then inserts can let two concurrent calls for the same slotId both return success, violating DUPLICATE_SLOT while passing this suite. Race two definitions with different capacities, require exactly one success and one DUPLICATE_SLOT, and verify the stored capacity matches the winner.

[ask-bonk]

P2 Duplicate expense IDs are never submitted concurrently. Two requests can both pass an existence check before either insert completes, both return success, and double-count the expense while all current checks pass. Race two expenses with the same ID and verify one succeeds, one returns DUPLICATE_ID, and totals include it only once.

[ask-bonk]

P2 Every monthlyTotals() assertion requests March, even though this setup seeds an April expense. An implementation that ignores the month argument and always returns March therefore passes the full eval. Query April as well and require its 90_000 total and payer row.

[ask-bonk]

Submitted 3 actionable inline review comments.

github run

[ask-bonk]

• P2 expense-ledger.eval.ts:351: After upgrade, March byPayer is not rechecked. The agent can preserve totalCents while breaking payer totals.
• P2 expense-ledger.eval.ts:431: Turn two no longer enforces the settlement transfer-count limit or rejects self-transfers, allowing a regressed settlement implementation to pass.
• P2 expense-ledger.eval.ts:400: budgetStatus() assertions permit arbitrary phantom category rows, despite the contract limiting rows to budgeted or spent categories.

github run

[ask-bonk]

P2 settlement() is only called after the fixtures have created multiple non-zero balances, and this check explicitly requires at least one transfer. An implementation that throws or returns a bogus payment when the ledger is empty/already settled therefore passes, even though the RPC must return payments that settle the current balances (which should be [] when nothing is owed). Exercise settlement() before adding expenses, or after constructing a zero-balance state, and require an empty transfer list.

[ask-bonk]

P2 monthlyTotals() is never queried for a valid month containing no expenses. An implementation that assumes an aggregate row exists and throws, or carries payer rows forward from another month, passes all current checks despite the contract requiring totals counting only that month. Query an unused month and require totalCents === 0 with an empty byPayer array.

[ask-bonk]

Submitted 2 actionable inline review comments.

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run