research: prove doctrine-shaped frozen-plan realization

[ezutfen]

Links Zutfen-LLC/inferswarm#50. This PR does not close the issue and must not be merged automatically; issue acceptance remains gated on review of the committed physical evidence.

Provenance

• Branch: poc/r1-frozen-plan-realization
• PR base branch: poc/n0-selective-model-block-loading
• Exact accepted fix: use current GPU for e4m3 capability FlashML-org/FreeToken#48 base SHA: 2d435c5b8addc4032fd1be3198d796e632648ac8 (merge commit of FreeToken PR research: prove accelerator residency without host mirror #15)
• Initial implementation commit: bc8e95514ee41b23f5741cf6b5e10870f00698de
• Review-fix commit: f8e9e8f7301e50ce1b42851ebd5e7191e3c71bfe
• No merge, rebase, or implementation import from the aborted N1 branch
• Model: nvidia/Qwen3.6-35B-A3B-NVFP4
• Revision: 491c2f1ea524c639598bf8fa787a93fed5a6fbce
• Fixture: canonical N0/fix: use current GPU for e4m3 capability FlashML-org/FreeToken#48 selective Block A, global layers 0–18

Frozen plan

• Research schema: inferswarm.r1.frozen-plan/1
• Plan version: r1-block-a-v1
• Canonical payload digest: sha256:430f84e406f51d037ab8ae2844865ff4d6b60d17a1d43d54180d509682ca2281
• Exact file SHA-256: 11bebfc891d078eafafeddc78a36bc7aba3943b97a849e02a72a480e55115062
• Refreshed result file SHA-256: e7d04bbafb9a26178690a589c65301a4e8918e880e61bd11a56a5d6b07cb484e
• The plan was frozen before retained physical execution and was not changed in response to physical outcomes.

The manifest records the Swarm/Node/Compute Unit/Memory Resources/backing/links, four Logical State Units, five materializations, roles and persistence, execution placement and required state, one mutable authority lineage, capacity expectations, and the forbidden persistent host mirror. It intentionally retains the exact N0 selective-key set so the pinned checkpoint selection is fully frozen.

These are internal research mappings and plain structures, not a final ExecutionPlan, resource schema, planner API, or model-strategy ABI. Generic validation/reconciliation contains no Qwen/MoE/expert/router/NVFP4/CUDA Graph vocabulary; the pinned mechanics live behind one narrow adapter.

Plan-driven realization

The runner only loads the frozen plan, constructs/verifies the known physical environment, calls realize_frozen_plan(...), executes repeated correctness decode, and writes evidence.

The generic realizer validates first, asks the adapter to realize each planned materialization, infers the release transition from:

required persistent final materialization realized
+ same-state host materialization role == staging/transient
-> release staging

The pinned adapter then invokes FlashML-org#48's existing detach_host_sources_for_full_residency() boundary. That boundary independently validates complete accelerator residency before detach. The runner contains no selective-load/populate/detach sequence and no R1 experiment-name branch. Plan interpretation ends at setup; decode uses the existing backend-native frozen device slot mapping.

Physical result

Committed status: R1_FROZEN_PLAN_REALIZATION_PASS.

Hardware/software:

• Host: inferswarm01
• Compute Unit: NVIDIA GeForce RTX 3060, UUID GPU-1fc28f83-1d45-926e-54d0-ba1e835ef099
• VRAM: 12,485,525,504 bytes; compute capability 8.6
• System RAM: 134,984,794,112 bytes
• Torch 2.11.0+cu130; CUDA 13.0

Plan validation rejected heavyweight execution unless the exact digest, model/revision, references, representations, authority, execution legality, forbidden persistence, and known capacity were valid. The plan declared 10,391,381,656 required persistent VRAM bytes, below the measured target capacity.

Observed reconciliation:

• planned required materializations missing: 0
• unplanned persistent materializations: 0
• resource/representation/byte/execution/authority mismatches: 0
• staging record: TRANSIENT_RELEASED_AS_PLANNED
• reconciliation passed: true

Memory accounting

• Persistent required: 10,391,381,656 bytes
  • non-routed device state: 1,689,347,736
  • routed accelerator banks: 8,636,596,224
  • mutable block-local runtime state: 32,718,848
  • correctness replay state: 32,718,848
• Persistent optional: 0 bytes
• Conservative transient/staging upper bound: 18,931,555,660 bytes (complete host source bank plus cumulative selective fetch bytes; not claimed as simultaneous measured residency)
• Host routed-state staging before release: 8,636,596,224 bytes
• Host routed-state staging after release and four decode runs: 0 bytes
• Unplanned persistent: 0 bytes
• Unexplained persistent host mirror: 0 bytes
• All 114 former host staging Tensor weak references were dead after release and repeated execution.
• The only persistent host evidence Tensor bytes were 327,680 bytes owned by the same-backend reference fixture; model/runtime host ownership roots were zero.

VmRSS, VmHWM, RssAnon, RssFile, RssShmem, VmSwap, CUDA allocated/peak, and page/swap counters are retained per checkpoint. Component ownership remains authoritative because the pinned allocator may retain freed raw pages without retaining a live Logical State Unit materialization.

Correctness and backend execution

• Required prefill/context lifecycle: exact
• Four repeated steady-state decode runs: exact
• Maximum absolute deviation: 0
• Maximum relative deviation: 0
• NaN count: 0
• Inf count: 0
• Post-release checkpoint/source/rematerialization sentinels: all 0
• Resident source-access attempts: 0
• Hot-path plan lookups: 0

The physical runner did not capture CUDA Graph decode. Graph compatibility is therefore claimed only as structurally preserved and regression-tested, not physically benchmarked.

Tests

• 74 passed: R1 validation/reconciliation tests, focused graph lifecycle regression, fix: use current GPU for e4m3 capability FlashML-org/FreeToken#48 resident-only lifecycle, canonical N0 selective block tests, and relevant ordinary offload tests.
• Ruff format/check: passed for all new Python files.
• Physical Block A evidence runner: passed.
• Frozen plan and result SHA-256 records: verified.
• Repository-wide pytest was attempted and remains blocked during collection by the existing tests/benchmarks versus top-level benchmarks namespace collision: tests/benchmarks/test_phase0_profile_provenance.py cannot import benchmarks.inferswarm_phase0. No unrelated test-layout changes are included.

Proof boundary

This proves that one predeclared doctrine-shaped plan caused the pinned runtime to validate, realize, audit, and correctly execute the exact intended resource/state/materialization arrangement without reintroducing an implicit persistent host mirror.

It does not prove automatic placement, candidate comparison, a planner optimizer, R2 split execution, multi-GPU/network/multi-node execution, elasticity, Adaptive Demand Profiles, health/quarantine, public strategy plugins, a final execution-plan schema, production discovery, non-NVIDIA support, or model generality.

Review fixes

• Staging release now requires a distinct successfully realized persistent final materialization, with the exact qualifying materialization IDs retained in observed evidence.
• Released staging retains and reconciles pre-release resource, representation, and byte evidence plus post-release zero state and exact released-byte accounting.
• Mutable authority observation is derived from realized runtime ownership rather than echoed from the frozen plan.
• New negative regressions cover missing final realization, released-staging reconciliation failures, and missing/wrong/lineage-mismatched authority.
• Physical R1 evidence was regenerated under the stricter audit semantics and passes with clean reconciliation.

[ezutfen]

R1 physical evidence looks strong, but I found three blocking correctness gaps in the generic frozen-plan realization/audit substrate. These matter because R1's result is the substrate itself, not merely the Block A happy path.

1. Staging release is not actually gated on an observed required persistent realization. In realize_frozen_plan(), realized_required_ids includes any PLANNED_AND_REALIZED record, including the transient staging record itself. final_exists then checks that the plan contains a required persistent candidate for the same logical state and that the state ID appears in realized_required_ids. Therefore staging can be released even when the persistent counterpart returned a failed/non-realized status. Fix the release predicate to require the specific required+persistent planned counterpart (or an equivalent legal final materialization) to have an observed PLANNED_AND_REALIZED record. Add a regression where staging realizes but the final persistent materialization does not; staging must not be released and realization must fail closed.

2. Released transient materializations bypass reconciliation of where/what was actually staged. reconcile_realization() immediately continues for TRANSIENT_RELEASED_AS_PLANNED, so a staging materialization can have the wrong actual representation/resource (or wrong pre-release byte count) and still reconcile cleanly once released. R1 requires intended-vs-observed reconciliation for staging too. Preserve/report the pre-release observed bytes (the existing released_bytes can serve this purpose) and still reconcile logical state, representation, resource, and expected staging bytes before accepting the released status. Add wrong-resource/wrong-representation/released-byte regressions.

3. The physical adapter does not independently realize/observe mutable authority. QwenBlockAResearchAdapter.observe_authorities() returns self.plan['authorities'] verbatim, so the observed authority side is tautologically equal to intended authority. R1 requires mutable authority to be part of observed realization, not just validated plan text. Establish/store the active authority as runtime realization state only after the planned mutable-authority materialization has actually been realized (an internal adapter field/callback is sufficient; no public API needed), and have observe_authorities() return that realized state. Add a regression proving missing/wrong authority establishment fails reconciliation.

The committed physical result otherwise matches the stated R1 claims: frozen digest/provenance, clean materialization happy-path reconciliation, host staging release, zero unexplained mirror, and exact repeated correctness. I would consider this mergeworthy after these three audit/realization holes are closed; regenerate the physical result if the result schema/reconciliation fields change.

[ezutfen]

Re-review at head f8e9e8f7301e50ce1b42851ebd5e7191e3c71bfe: the three blocking R1 audit findings are resolved.

• Staging release now requires a distinct required persistent materialization for the same Logical State Unit that is actually observed live and matches its planned resource, representation, and bytes. Missing/not-realized final residency leaves staging unreleased and fails reconciliation.
• TRANSIENT_RELEASED_AS_PLANNED now retains and reconciles pre-release bytes/resource/representation, post-release zero bytes, released-byte accounting, lifecycle, and the exact qualifying final materialization IDs.
• Mutable authority is now recorded only when the mutable runtime materialization is physically realized; observe_authorities() returns runtime-owned observations rather than echoing the plan, and reconciliation verifies that authority is backed by a live realized owner.

The regenerated physical result keeps the frozen-plan bytes/digest unchanged, has clean reconciliation, 0 unplanned persistent bytes, 0 unexplained persistent host mirror bytes, exact prefill + four decode repetitions, and all staging/source sentinels clean. The focused suite is reported as 74 passed; the repository-wide collection limitation remains the pre-existing benchmarks namespace collision.

No remaining blocker found. PR #16 is mergeworthy for the stated R1 proof boundary. I am not merging it in this review.