Define structured triage and calibration checklists in schema.json, update Reviewer/Calibrator skills, add Patcher safety guards, and document pipeline assumptions

- Add triage_checklist (12 validity constraints) to schema.json and mantis-review/SKILL.md.
- Add calibration_checklist (27 sanity caps) to schema.json and mantis-calibrate/SKILL.md.
- Add other recommended schemas to schema.json (kb_index, threat_model, tx_log_entry, repro_attempts, report_summary, execution_log_entry).
- Improve finding validation rules in schema.json (re-attack validation, exploit chain bypass check).
- Fix duplicate_of schema to not allow null values when status is DUPLICATE.
- Add path and Cwd safety guards to mantis-patch/SKILL.md to prevent accidental usage of original workspace.
- Document static codebase assumption in README_AGENTS.md.
- Document exploit chain reproduction limitation in README_AGENTS.md.
- Document patch verification and re-attack constraints in README_AGENTS.md.
- Add Continuous Pipeline item to Roadmap in README.md.

TAG=agy
CONV=56291245-d9db-4780-b260-5992eb52e24e

Change-Id: Ia34008450c35c956a1e522eb38690590496b5790

Author: nickg-ca

README.md

@@ -229,6 +229,11 @@ skill performance, see the [Agent Reference Guide](README_AGENTS.md).
 
 ## Roadmap / Future Work
 
+*   **Continuous Pipeline:** The current pipeline is designed to be run as a
+    point-in-time review of a codebase, and not as something that is intended to
+    regularly sync with upstream changes mid-run. It should be straightforward
+    (if a little tricky) to tweak the pipeline to better support this, but it
+    probably will not work today.
 *   **Skill Self-Improvement (Meta-Learning):** The current
     `workspace/learnings.jsonl` and Knowledge Base (KB) architecture tracks
     codebase-specific empirical outcomes to adapt the `THREAT_MODEL.md` and

README_AGENTS.md

@@ -42,6 +42,15 @@ The Mantis Skills suite is designed as a modular, decoupled set of tools that
 can be executed sequentially or in parallel. Each stage reads and writes to a
 shared state stored on disk.
 
+> [!IMPORTANT] **Static Codebase Assumption:** The pipeline is designed to run
+> against a **single, static version (snapshot) of the codebase** at a time. It
+> is not designed as a continuous process that watches a live codebase or
+> handles concurrent modifications to the source code files by external
+> processes during execution. If the source files are modified during a run, it
+> may lead to inconsistent findings, broken line references, or failed patch
+> applications. Rescans should be initiated as separate, distinct runs (e.g.,
+> triggered by a new commit or changelist).
+
 ```mermaid
 graph TD
     subgraph CoreStages [Pipeline Execution Loop]
@@ -263,6 +272,45 @@ deterministic execution, you can build a pipeline that:
 
 --------------------------------------------------------------------------------
 
+## Exploit Chains and Reproduction Limits
+
+The Mantis pipeline supports identifying complex, multi-step exploit chains (via
+`/mantis-chain`), but **it does not attempt to programmatically write or execute
+end-to-end reproduction scripts for these chains**.
+
+*   **Constituent Reproduction Only:** The pipeline only reproduces the
+    individual, constituent findings.
+*   **Static Confirmation:** An exploit chain finding is marked as
+    `statically_confirmed` if all its constituent findings have been
+    successfully reproduced individually.
+*   **Simplification Decision:** This is an intentional design decision to limit
+    complexity, as automated multi-stage exploit orchestration is highly
+    environment-dependent. Users wishing to verify end-to-end chains must write
+    custom orchestrators or manually verify the combined flow.
+
+--------------------------------------------------------------------------------
+
+## Patch Verification and Re-attack Constraints
+
+The `schema.json` contract defines validation rules for findings that have been
+patched. While `/mantis-patch` is designed to verify patches using a re-attack
+step (confirming `reattack_status`), the schema **does not strictly require
+re-attack fields** (`reattack_status`, etc.) even when `patch_status` is
+`VERIFIED_SECURE`.
+
+This is an intentional design decision to support:
+
+*   **Binary-only targets:** Compiled binaries or firmware blobs skip code
+    modification and re-attack testing, instead providing a high-level
+    mitigation recommendation in the `patch_diff` field. These may still be
+    marked as resolved/secure without undergoing a functional re-attack.
+*   **Verification Fallbacks:** If the re-attack tool execution fails (due to
+    timeouts or sandbox infrastructure issues), the pipeline can still output
+    the generated patch that passed the initial post-patch verification run,
+    rather than failing validation entirely.
+
+--------------------------------------------------------------------------------
+
 ## The Reality of Non-Determinism
 
 A critical concept to understand when using AI for security research is

mantis-calibrate/SKILL.md

@@ -32,8 +32,8 @@ produce a final risk score (1-10).
     -   Updates finding files in-place with scoring/calibration fields
         (`impact_score`, `likelihood_score`, `availability_tier`,
         `inferred_exposure`, `attacker_position`, `mantis_risk_score`,
-        `priority`, `sanity_triage_applied`, `outrage_commentary`,
-        `executive_summary`).
+        `priority`, `sanity_triage_applied`, `calibration_checklist`,
+        `outrage_commentary`, `executive_summary`).
     -   Reusable helper script `workspace/helpers/append_calibrate.py`.
 -   **Preconditions**:
     -   Confirmed or raw findings must exist in `workspace/findings/`.
@@ -607,8 +607,50 @@ Execute the calibration as follows:
     -   `"priority"` (CRITICAL, HIGH, MEDIUM, LOW)
     -   `"sanity_triage_applied"` (semicolon-separated list of Section 3 rules
         that fired, most-restrictive first, or null)
+    -   `"calibration_checklist"` object containing evaluations for all 27
+        sanity caps (each key in the object maps to the sanity cap rule of the
+        matching name from Section 3 above):
+
+        ```json
+        {
+          "repro_failure": { "fires": <bool>, "reason": "<string>" },
+          "unreachable_inputs": { "fires": <bool>, "reason": "<string>" },
+          "third_party_reachability": { "fires": <bool>, "reason": "<string>" },
+          "minor_config_hygiene": { "fires": <bool>, "reason": "<string>" },
+          "non_security_critical": { "fires": <bool>, "reason": "<string>" },
+          "vague_code_paths": { "fires": <bool>, "reason": "<string>" },
+          "unreliable_triggers": { "fires": <bool>, "reason": "<string>" },
+          "prerequisite_shell": { "fires": <bool>, "reason": "<string>" },
+          "physical_long_term": { "fires": <bool>, "reason": "<string>" },
+          "trusted_controller_zero_delta": { "fires": <bool>, "reason": "<string>" },
+          "standard_host_attacks": { "fires": <bool>, "reason": "<string>" },
+          "static_confirmation": { "fires": <bool>, "reason": "<string>" },
+          "strict_xss": { "fires": <bool>, "reason": "<string>" },
+          "internal_nested": { "fires": <bool>, "reason": "<string>" },
+          "probabilistic_llm": { "fires": <bool>, "reason": "<string>" },
+          "supply_chain_prerequisites": { "fires": <bool>, "reason": "<string>" },
+          "non_default_config": { "fires": <bool>, "reason": "<string>" },
+          "confidential_computing_host": { "fires": <bool>, "reason": "<string>" },
+          "trusted_controller_critical_bypass": { "fires": <bool>, "reason": "<string>" },
+          "local_attack_vector": { "fires": <bool>, "reason": "<string>" },
+          "self_contained_blast": { "fires": <bool>, "reason": "<string>" },
+          "rarely_exposed": { "fires": <bool>, "reason": "<string>" },
+          "equivalent_primitives": { "fires": <bool>, "reason": "<string>" },
+          "documented_insecure_config": { "fires": <bool>, "reason": "<string>" },
+          "physical_temporary": { "fires": <bool>, "reason": "<string>" },
+          "high_privilege_external": { "fires": <bool>, "reason": "<string>" },
+          "trusted_controller_standard_bypass": { "fires": <bool>, "reason": "<string>" }
+        }
+        ```
+
+        For each rule, `fires` must be `true` if the sanity cap rule applies
+        (fires) to this finding, capping or downgrading its score/priority, or
+        `false` if it does not apply. The `reason` must describe the evaluation.
+
     -   `"outrage_commentary"` (your reasoning about the outrage factor)
+
     -   `"executive_summary"`
+
     -   An entry to the `"history"` array:
 
         ```json

mantis-patch/SKILL.md

@@ -133,6 +133,14 @@ Execute the patching and verification stage as follows:
                 location (e.g., `/tmp/mantis-shadow-[finding_id]/`).
             -   Perform all edits, compilation, and reproduction testing inside
                 this temporary shadow directory.
+            -   **Critical Guard (Path and Working Directory Safety):** You must
+                ensure that every command executed (compilation, testing,
+                verification) runs with its working directory (`Cwd`) explicitly
+                set to the shadow directory. If the finding's `run_command`
+                contains absolute paths to the original workspace, you must
+                rewrite them to point to the corresponding paths in the shadow
+                directory before execution. Do not execute any modification or
+                verification commands against the original workspace.
             -   Generate the unified patch diff by comparing the original source
                 files in the workspace with the modified files in the shadow
                 directory.
@@ -160,18 +168,26 @@ Execute the patching and verification stage as follows:
     `"repro_file_path"` and `"run_command"` from the reproduction entry to
     verify the patch.
 
+    *   **Cwd Enforcement:** You must execute the reproducer script with the
+        working directory (`Cwd`) set to the shadow directory (if using Option
+        A). Ensure the command targets the copy in the shadow directory, not the
+        original workspace.
+
     -   **VERIFIED SECURE:** If the post-patch sandbox run fails to reproduce
         the bug, the initial patch holds. However, you must now perform a
         **Re-attack**: assume the patch is flawed and explicitly attempt to
         write a new reproducer variant that bypasses your patch to reach the
         same root cause. Only if the re-attack also fails to bypass the fix
         should you mark the patch as fully successful! To ensure true
         independence, launch a fresh `@mantis-reproduce --reattack` subagent
-        against the patched code directory to perform this re-attack. The
+        against the patched code directory to perform this re-attack.
+        **Important:** Ensure you explicitly pass the shadow directory path (not
+        the original workspace) as the target directory for the subagent. The
         reproducer agent running with `--reattack` will write its outcomes
         directly into the primary finding's `reattack_status`,
         `reattack_file_path`, `reattack_run_command`, and `reattack_output`
         fields on disk, keeping the initial `repro_*` fields untouched.
+
     -   **VERIFICATION FAILED:** If the sandbox execution still triggers the
         bug, or if your re-attack successfully bypasses your patch, the patch is
         insufficient. Re-evaluate and adapt your fix.

mantis-review/SKILL.md

@@ -27,7 +27,7 @@ to verify validity and filter out noise and false positives.
     -   Target source code files (at paths/lines in `code_paths`).
 -   **Writes**:
     -   Updates findings on disk in-place (sets `"status"`, `"reasoning"`,
-        `"repro_hints"`, and appends history).
+        `"repro_hints"`, `"triage_checklist"`, and appends history).
     -   Writes helper script `workspace/helpers/append_review.py`.
 -   **Preconditions**:
     -   `workspace/findings/` exists with finding files.
@@ -124,6 +124,7 @@ Execute your validation as follows:
         resources on hallucinated bugs.
 
     -   **Status Resolution:**
+
         -   Mark as **FALSE_POSITIVE** if it violates any of the 12 rules above.
         -   Mark as **VALID** if it passes all rules and has a clear,
             triggerable flaw.
@@ -133,6 +134,15 @@ Execute your validation as follows:
         -   Mark as **NEEDS_RESEARCH** if the review is inconclusive due to high
             complexity, unresolved external APIs, or massive call graphs.
 
+    -   **Checklist Construction:**
+
+        -   Construct the `triage_checklist` object evaluating all 12 negative
+            constraints. For each rule, set `passes` to `true` if the finding
+            satisfies the constraint (i.e. it does not violate the rule, meaning
+            the bug remains potentially valid under that rule), or `false` if
+            the finding violates the rule (which requires it to be marked as
+            `FALSE_POSITIVE`).
+
 4.  **Construct Reproduction Script Hints:** For every finding marked as
     **VALID** or **PROVISIONALLY_VALID**, provide high-signal `"repro_hints"`
     explaining how a reproducer agent can trigger the bug, what inputs or
@@ -155,6 +165,32 @@ Execute your validation as follows:
     -   A `"reasoning"` field.
     -   A `"repro_hints"` field (optional for `"NEEDS_RESEARCH"` or
         `"FALSE_POSITIVE"`).
+    -   A `"triage_checklist"` object containing evaluations for all 12 negative
+        constraints (each key in the object maps to the constraint of the
+        matching name from Section 3 above):
+
+        ```json
+        {
+          "ignore_hypothetical_misuse": { "passes": <bool>, "reason": "<string>" },
+          "ignore_missing_hygiene": { "passes": <bool>, "reason": "<string>" },
+          "require_strict_reproducibility": { "passes": <bool>, "reason": "<string>" },
+          "avoid_pedantic_linting": { "passes": <bool>, "reason": "<string>" },
+          "no_security_flaw_stretching": { "passes": <bool>, "reason": "<string>" },
+          "evaluate_questionable_file_paths": { "passes": <bool>, "reason": "<string>" },
+          "ignore_resource_exhaustion_dos": { "passes": <bool>, "reason": "<string>" },
+          "intrinsic_security_flaws": { "passes": <bool>, "reason": "<string>" },
+          "verify_mitigations_pragmatically": { "passes": <bool>, "reason": "<string>" },
+          "refine_code_paths_strictly": { "passes": <bool>, "reason": "<string>" },
+          "ignore_simd_vector_padding": { "passes": <bool>, "reason": "<string>" },
+          "ensure_source_code_coherence": { "passes": <bool>, "reason": "<string>" }
+        }
+        ```
+
+        For each rule, `passes` must be `true` if the finding satisfies the
+        validity constraint (i.e. it is NOT ruled out by the constraint), or
+        `false` if it violates the constraint (ruling it out). The `reason` must
+        describe the evaluation.
+
     -   An entry to the `"history"` array:
 
     ```json