docs: add a status and doctor troubleshooting matrix - #50
Conversation
status and doctor emit a lot of fields and no guidance on which combinations mean something is wrong. Add a matrix plus a worked section per state, each with the next safe action. Covers the five states the issue names -- plugin installed with no observed evidence, Shadow Mode active, incomplete hook coverage, enforcement not earned, and stale or drifted evidence -- and adds CODEX_NOT_FOUND, because that one makes every status number meaningless and should be checked first. Example output was generated by running both commands against a scratch data root rather than written by hand, so the field names and shapes match what diagnostics.py actually emits. Identifiers were then replaced with synthetic placeholders; no real repository hash appears. The distinctions worth having in writing, all of which read the wrong way round at first glance: - capability: Tool Enforcement appears on a fresh install. It is what the integration can do, not what is in force; authority.effective is that. - an empty effective_blockers does not mean enforcement is happening, only that nothing blocks the configured mode -- which is shadow. - doctor reports capability_label: Observe when it cannot confirm control, degrading to what it can prove. - when ledger.valid is false, status falls back to summarizing raw records, so counts stay populated while EVIDENCE_ROOT_UNVERIFIED blocks promotion. Populated counts are not verified evidence. Nothing implies enforcement where the runtime fails open: the doctor section states that effective_policy.effective false means actions proceed. Documentation only. Linked from docs/index.md under Operations, and links out to the Codex integration, the integration labels and the privacy docs. Closes SignalLayerLabs#35
There was a problem hiding this comment.
Thanks — this is a useful addition and the overall structure is strong. I found three semantics that need tightening before merge:
coverage_ratio currently summarizes the decision evidence MARGINAL actually observed. It does not prove total Codex runtime coverage or directly measure actions that never reached the hooks. Please avoid describing < 1.0 as “actions Codex took were not seen by a hook,” and make that limitation explicit.
stale_session_receipts do not currently expire on a time-based TTL. Unreachable receipt files are counted as stale, but they do not simply “age out.” Please adjust that wording to reflect the current liveness behavior.
CODEX_NOT_FOUND does not make existing status evidence meaningless. status can still report previously persisted local evidence even when Codex is currently unavailable. doctor tells us whether the integration can observe new Codex activity right now. Similarly, hook_state: not_observed plus zero evidence alone does not prove that the plugin is installed.
The rest looks good, especially the distinction between capability and effective authority, the Shadow Mode explanation, and the fail-open wording.
Once these three points match the runtime semantics, this should be good to merge.
All three were me asserting more than the code supports. coverage_ratio. Both sides of covered_actions / coverable_actions are sums over decision records already in the evidence store, so an action that never reached a hook produces no record and lands in neither -- it is invisible to the ratio, not counted against it. "Actions Codex took were not seen by a hook" described the one thing the number cannot see. Reworded to what it does summarize, with the limit stated outright: 1.0 is equally consistent with complete coverage and with a hook that stopped firing, and doctor is the tool for that question. Also noted that the 0.0 on a fresh install is the zero-denominator fallback in diagnostics.py, not a measurement of zero coverage. stale_session_receipts. There is no TTL. _active_hook_sessions counts a receipt stale when _live_session_repository returns None -- unreadable file, symlink, oversized, non-loopback host, token under 16 bytes, or a loopback probe that does not come back ok inside 0.25s. "They age out" said the count resolves by waiting; it resolves by removing the file. Added the two consequences that follow: a slow session is counted stale for that run, and stale is not filtered by repository the way active_hook_sessions is. CODEX_NOT_FOUND. status reads evidence persisted on disk and keeps reporting it whether or not the Codex CLI is on PATH right now, so "status numbers are meaningless until this clears" was wrong. Split the two questions explicitly -- doctor asks whether new activity can be observed, status reports what was already recorded -- and named the failure mode that actually applies: reading an unchanging status as a quiet period rather than a broken integration. Same correction on the first matrix row: not_observed with zero evidence is the absence of evidence, and an uninstalled plugin prints exactly the same thing, so it cannot be described as "the plugin is installed".
|
All three corrected in 8051fe2 — you were right on each, and each one was me asserting more than the code supports.
Your fourth point took the first matrix row with it. Docs-only, no code touched. |
Closes #35.
Adds
docs/operations/troubleshooting.md— a matrix plus a worked section per state, each ending in the next safe action. Linked fromdocs/index.mdunder Operations.Acceptance criteria
CODEX_NOT_FOUND, because it makes everystatusnumber meaningless and should be checked first.The examples are real output, not invented
I generated them by running both commands against a scratch data root:
so every field name and shape matches what
diagnostics.pyactually emits —hook_state,coverage_ratio,next_promotion_blockers,ledger.first_invalid_sequence,effective_policy.effective. Identifiers were then replaced with placeholders likerepo-alpha…; I checked the file for 64-hex strings and there are none, so no real repository hash appears.Fail-open is stated explicitly
The
doctorsection says plainly thateffective_policy.effective: falsemeans the runtime fails open and actions proceed, and that nothing instatusshould be read as blocking while that holds.Related, and the reason I wrote that section at all: an empty
effective_blockersreads like "nothing is stopping enforcement" when it actually means "nothing blocks the configured mode" — and the configured mode isshadow. The page tells the reader to readconfigured_modeandeffectivetogether.Three other things that read the wrong way round
Each has a short note in the page:
capability: Tool Enforcementshows on a fresh install with zero evidence. It is what the integration can do;authority.effectiveis what is in force.doctorreportscapability_label: Observewhen it cannot confirm control, even though Codex is Tool Enforcement when working — the label degrades to what can be proven.ledger.validisfalse,statusfalls back to summarizing raw records, so the counts stay populated whileEVIDENCE_ROOT_UNVERIFIEDblocks promotion. The page says not to read populated counts as verified evidence.On coverage, the page frames
coverage_ratiobelow1.0as expected rather than broken — it is the same reason Codex is labelled Tool Enforcement rather than Full Compute Enforcement — and says to treat a sudden drop as the signal, not any value below 1.29 relative links across the touched files checked; none broken.