Add tiered iterative reproduce strategy and retries

TAG=agy

CONV=dc7c7ce1-1d98-4b34-8b02-6b6c97c4523d

Change-Id: Id5383c205c389ff6b6d6413572d38d050f540a22

Author: nickg-ca

README_AGENTS.md

@@ -164,49 +164,64 @@ when unavailable. Consumers query via
     vulnerabilities from `workspace/historical_learnings.jsonl` to enrich
     summaries and provide a quick reference map to optimize downstream planning
     and research.
+
 04. **`/mantis-architecture` (Knowledge Base Architect):** Analyzes the codebase
     and clears the `workspace/learnings.jsonl` inbox to synthesize a permanent,
     interlinked Markdown Knowledge Base (`workspace/kb/`) detailing entities,
     data flows, and historical vulnerability classes.
+
 05. **`/mantis-threat-model` (Threat Modeler):** Evaluates the entities and
     architecture defined in the KB to establish or refine a living
     `workspace/kb/THREAT_MODEL.md`, focusing on trust boundaries and attacker
     profiles.
+
 06. **`/mantis-plan` (Strategist):** Scans workspace boundaries and reads the KB
     indices to output a targeted review strategy into `workspace/plan.json`,
     injecting specific `kb_references` file paths for context.
+
 07. **`/mantis-researcher` (Mantis Researcher):** Executes file-by-file triage
     and deep security flaw reviews, outputting hotspots as individual JSON files
     in `workspace/findings/`.
+
 08. **`/mantis-dedupe` (Deduplicator):** Groups index-based duplicate findings,
     merging records and deleting redundancies within `workspace/findings/`.
+
 09. **`/mantis-review` (Validator):** Filters out false positives using strict
     pragmatic constraints, updating the status in
     `workspace/findings/<id>.json`.
+
 10. **`/mantis-critic` (Critic):** Verifies release-build crash reproducibility
     (ignoring debug/assert checks), updates production viability in
     `workspace/findings/<id>.json`, and appends false positives/non-viable paths
     to `workspace/learnings.jsonl`.
+
 11. **`/mantis-reproduce` (Proof-of-Concept Developer):** Writes
     Proof-of-Concept Reproduction Scripts (Repros) or raw payloads, executes
-    them in isolated environments such as gVisor or Virtual Machines, and
-    updates reproduction status in `workspace/findings/<id>.json`.
+    them using a Tiered Iterative Reproduction strategy (unit micro-harness ->
+    functional subsystem -> full sandboxed service) with
+    intra/inter-conversation retries in isolated environments (gVisor, VMs,
+    QEMU), and updates reproduction status in `workspace/findings/<id>.json`.
+
 12. **`/mantis-chain` (Vulnerability Chainer):** Analyzes individual validated
     findings and knowledge base primitives to identify and construct complex
     multi-step exploit chains, creating new "Super Findings" in
     `workspace/findings/`.
+
 13. **`/mantis-patch` (Patcher):** Generates and applies code fixes, runs
     post-patch validation tests inside the sandbox, updates patch status in
     `workspace/findings/<id>.json`, and appends logs to
     `workspace/learnings.jsonl`.
+
 14. **`/mantis-calibrate` (Risk Calibrator):** Calculates a final numerical
     Mantis Risk Score (1-10) for each finding in the workspace directory based
     on impact, evidence, and viability, appending the results directly to each
     `workspace/findings/<id>.json` file.
+
 15. **`/mantis-reflect` (Reflector):** Parses the execution trajectories of the
     agents from the current round, extracting false assumptions, tool failures,
     and successes, and appends these structured insights to the
     `workspace/learnings.jsonl` inbox.
+
 16. **`/mantis-report` (Reporter):** Generates a human-readable security review
     packet containing verified/reproduced findings, evidence, risk rationales,
     and patch information at `workspace/report/review_packet-latest.md` (and

mantis-critic/SKILL.md

@@ -245,7 +245,11 @@ Execute the critic evaluation as follows:
    CODE_ROOT and read at least **15 lines of preceding context** and **15 lines
    of succeeding context** around the designated line numbers. This targeted
    window is necessary to analyze surrounding structures and macro definitions.
-   Proceed to Steps 4-5.
+   Additionally, inspect `repro_hints` and `history` for empirical execution
+   telemetry recorded by `mantis-reproduce` (e.g. `build_profile`,
+   `sanitizers_used`, `assertions_disabled`, `ingress_blocked`). Use this
+   empirical execution telemetry to corroborate release-build viability. Proceed
+   to Steps 4-5.
 
 4. **Evaluate Domain-Specific Viability Constraints:**
 

mantis-pipeline-adapter/SKILL.md

@@ -105,6 +105,10 @@ Follow these guidelines during the consultation:
     LLM reasoning, runs after the snapshot is pinned and before the first
     code-reading analysis stage, and degrades gracefully to grep when
     unavailable.
+11. **Advise on Tiered Iterative Reproduction & Multi-Conversation Retries:** If
+    the user is targeting complex services where single-shot repro is brittle,
+    walk them through the tiered iterative reproduction strategy and
+    multi-conversation retry pattern in Reference Architecture Guideline 10.
 
 ## Reference Architecture Guidelines
 
@@ -1069,3 +1073,77 @@ behave exactly as they do today:
   slices) by using function boundaries from the structural index.
 - The structural index and the RAG index can share the same vector embedding
   infrastructure if both are implemented.
+
+______________________________________________________________________
+
+### 10. Tiered Iterative Reproduction & Multi-Conversation Retry Strategy
+
+For complex services, attempting a single-shot reproduction directly against a
+full sandboxed service often suffers from high search entropy, brittle
+configuration, and hard-to-debug failures. A tiered strategy breaks reproduction
+into incremental milestones, while an inter-conversation retry architecture
+prevents reasoning deadlocks and context bloat.
+
+#### A. Orchestration & Inter-Conversation Retries
+
+1. **Intra-Conversation Retries (Local Agent Trajectory):**
+
+   - The active subagent conversation retries 2–3 times locally within its
+     context window to adjust parameters, fix setup bugs, or refine payloads.
+
+2. **Inter-Conversation Retries (Fresh Context + Accumulated Artifacts):**
+
+   - **Trigger:** If intra-conversation retries fail to reach Tier 3
+     (`reproduced`), the orchestrator terminates the stalled conversation and
+     launches a **new subagent conversation** (a fresh context window).
+   - **Context Provisioning:** The orchestrator populates the new prompt with
+     structured attempt data from
+     `state_root/workspace/archive/.repro_attempts.json` and trajectory
+     learnings from `workspace/learnings.jsonl` (e.g., *"Attempt 1 failed due to
+     missing auth header X; Attempt 2 proved parser strips unescaped quotes"*).
+   - **Benefit:** Eliminates context bloat and reasoning inertia ("hallucination
+     traps"), enabling a fresh agent to solve the problem using prior empirical
+     observations without repeating past mistakes.
+
+3. **Attempt Cap Accounting & Arithmetic:**
+
+   - The orchestrator maintains
+     `state_root/workspace/archive/.repro_attempts.json`.
+   - **Tier 3 Increment Only:** Only Tier-3 full sandboxed service executions
+     (or full end-to-end reproducer runs) increment the per-finding attempt
+     counter toward the absolute hard ceiling of 6.
+   - **Stepping-Stone Sub-Budget:** Internal Tier-1 and Tier-2 trial runs are
+     bounded by a local sub-budget (max 3 trial executions per conversation) and
+     do not consume the absolute 6-attempt cap.
+
+#### B. Harness-Enforced Tier 4 (Staging & Live Pre-Production Execution)
+
+To prevent un-gated exploit execution on live infrastructure, Tier 4 (Staging /
+Pre-Production verification) is **strictly owned and enforced by the
+programmatic orchestrator harness**, never by LLM discretion:
+
+- **Sandbox Boundary:** `mantis-reproduce` executes strictly within isolated
+  local sandboxes (Tiers 1–3) ending at Tier 3 (`reproduced` /
+  `failed_to_reproduce`).
+- **Deterministic Gate:** The harness intercepts a Tier 3 `reproduced` verdict.
+  If live/staging validation (Tier 4) is configured, the harness **MUST NOT**
+  automatically invoke remote execution. It must enforce a **programmatic
+  Human-in-the-Loop gate**:
+  - Prompt the human operator for explicit interactive confirmation, OR
+  - Require a cryptographically signed approval token / authorization callback.
+- **Fail-Closed Default:** If human approval is missing or denied, Tier 4 is
+  skipped and the Tier 3 sandboxed verdict remains authoritative.
+
+#### C. Local Ingress / Middlebox Edge Annotation (Post-Tier-3 Sandbox Verification)
+
+To eliminate false-positive findings caused by default edge filters ("Works on
+localhost:8080, but dies at the WAF/proxy"), the orchestrator can optionally
+execute a Tier 3 PoC through a local reverse proxy or API gateway (e.g. NGINX,
+Envoy, ModSecurity) running inside the local sandbox:
+
+- **Annotation Only:** A payload blocked by a local middlebox MUST NOT downgrade
+  a Tier 3 `reproduced` verdict to `failed_to_reproduce`.
+- **Critic Integration:** The harness records `ingress_blocked: true` (or edge
+  filter details) in the finding's `repro_hints` or history. This provides
+  empirical evidence for `/mantis-critic` to classify `production_viability` as
+  `"CONDITIONAL_VIABLE"` (mitigated by default edge proxy configuration).

mantis-reproduce/SKILL.md

@@ -198,6 +198,18 @@ Execute the reproduction stage under these constraints:
          viability, but always check status).
      - If no applicable findings exist, notify the user and exit.
 
+   **Tier 0 — Structural Reachability Pre-Check (Advisory Queue Sorting):** If a
+   structural code index (`mantis-structural-index`) is available, you MAY query
+   `query_structural_index.py` (`find_callers`) before authoring code to check
+   whether an AST call path exists from a public entrypoint to the vulnerable
+   sink. Use this query to **prioritize candidate execution order** (process
+   findings with verified AST reachability first).
+
+   - **CRITICAL HINT-ONLY GUARDRAIL:** AST reachability is a ranking HINT ONLY.
+     Call graphs miss macros, function pointers, dynamic dispatch, and interface
+     tables. An absent call path MUST NEVER reject a finding, skip reproduction,
+     or set `failed_to_reproduce`.
+
    **Snapshot drift check:** For each loaded finding, if it already has a
    `repro_snapshot_id` and Block B (Step 0) returns NOT_MATCHED, treat any
    stored PoC/offsets as STALE: regenerate the reproducer from scratch against
@@ -285,11 +297,57 @@ Execute the reproduction stage under these constraints:
    standard execution. Use your best judgment to construct a working harness for
    the artifact.
 
-   - *Optional Parallel Trajectory Search:* If your environment or agent
-     framework supports spawning subagents, you can deploy multiple concurrent
-     agents to attempt writing the reproducer via different logical approaches.
-     If any trajectory succeeds, immediately adopt its payload and discard the
-     others to escape potential "give up" loops.
+   - *Parallel Trajectory Search vs. Tiered Iterative Reproduction:*
+
+     - **Parallel Trajectory Search (Breadth-First):** When subagents are
+       available, deploy concurrent workers taking diverse logical approaches to
+       reproduce the bug. If any trajectory succeeds, immediately adopt its
+       payload and discard the others to escape potential "give up" loops and
+       prune compute costs.
+
+     - **Tiered Iterative Reproduction (Depth-First Payload Refinement):** Each
+       trajectory worker (or a single agent) uses a tiered escalation ladder
+       (Tier 1 -> Tier 2 -> Tier 3) to refine its trigger payload incrementally
+       rather than attempting a single-shot end-to-end launch.
+
+   - *Tiered Iterative Execution Ladder:*
+
+     - **Tier 1 (Micro-Harness / Sink Logic Validation):** Construct a
+       lightweight test calling the vulnerable function/module directly to
+       verify that the core bug hypothesis is sound in isolation.
+     - **Tier 2 (Subsystem / Interface Validation):** Pass the payload through
+       input serialization, parsers, routing, and auth wrappers to verify the
+       input survives intermediate processing without sanitization or
+       truncation.
+     - **Tier 3 (Full Sandboxed Service / E2E Validation):** Execute the
+       self-contained PoC against the target service via public APIs inside the
+       isolated sandbox (Docker, QEMU, VM). Yields the authoritative
+       `reproduced` verdict per Block F.
+
+   - **CRITICAL STEP-4 TIER-1 HARD GATE (Fail-Closed):**
+
+     - Tiers 1 and 2 are **internal stepping stones only**. You **MUST NEVER**
+       record `repro_status = "reproduced"` or `"statically_confirmed"` based on
+       a Tier-1 or Tier-2 execution.
+     - If a crash can **ONLY** be achieved by compiling a direct-call harness
+       that feeds a private/static function or bypasses the public API (Step 4),
+       and the payload cannot be escalated to trigger through Tier 3 (the public
+       API / sandboxed service), you **MUST TERMINATE AND RECORD**
+       `repro_status = "failed_to_reproduce"` with details citing
+       `"Internal Invariant Protection"`.
+
+   - *Attempt Cap Accounting & Local Retries:*
+
+     - **Sub-Tier-3 Stepping-Stone Sub-Budget:** Internal Tier-1 and Tier-2
+       trial runs are bounded local execution steps (max 3 trial executions per
+       conversation) and **DO NOT** increment the absolute per-finding attempt
+       counter in `state_root/workspace/archive/.repro_attempts.json`.
+     - **Absolute Attempt Cap Counting:** Only Tier-3 full sandboxed service
+       executions (or full end-to-end reproducer runs) increment the absolute
+       attempt counter toward the hard ceiling of 6 (Section 6).
+     - **Intra-Conversation Retries:** When a tier fails, inspect logs, adjust
+       payload parameters, fix harness setup, and retry up to 2-3 times within
+       the active conversation before reporting back to the orchestrator.
 
 ### Step 3a: Variant Hunting (re-attack only, MANDATORY)
 
@@ -534,17 +592,25 @@ and apply INV-1 (downgrade `VERIFIED_SECURE` → `VERIFICATION_FAILED`).
      * Open the lock file `state_root/workspace/archive/.repro_attempts.lock`
        (creating it if missing) and acquire an exclusive lock (`fcntl.flock`
        with `fcntl.LOCK_EX`) inside a context manager (`with` statement).
+
      * Read the current contents of the cache file
        `state_root/workspace/archive/.repro_attempts.json` (treating it as `{}`
        if missing or empty).
+
      * Increment the `count` field of this finding's cache-key entry — keyed by
        `signature` if present, else `stable_key`, the SAME key selection defined
-       above (the `{count, last_snapshot}` object) — by 1.
+       above (the `{count, last_snapshot}` object) — by 1 ONLY for Tier-3 (full
+       end-to-end sandboxed service) executions. Internal Tier-1 and Tier-2
+       stepping-stone trials MUST NOT increment `count` (they are governed by
+       the sub-budget rule in Step 3).
+
      * Write the updated JSON to a temporary file in the same directory (e.g.,
        `state_root/workspace/archive/.repro_attempts.json.tmp`).
+
      * Atomically replace the target cache file with the temporary file (e.g.,
        `os.replace` in Python) to ensure readers never see a truncated or
        incomplete file.
+
      * Close the lock file descriptor to release the lock (automatically handled
        by exiting the `with` context manager).
 
@@ -564,6 +630,11 @@ and apply INV-1 (downgrade `VERIFIED_SECURE` → `VERIFICATION_FAILED`).
 
      - `"repro_snapshot_id"`: the current SNAPSHOT_ID this run executed against.
 
+     - `"repro_hints"`: Record compilation and sandbox execution telemetry
+       (e.g., `sanitizers_used: ASan+UBSan`, `assertions_disabled: true`,
+       `build_profile: release`) to provide empirical execution evidence for
+       `/mantis-critic`.
+
      - If reproduction succeeds (`repro_status` is evaluated as `"reproduced"`
        or `"statically_confirmed"`) and the finding's current `"status"` is
        `"PROVISIONALLY_VALID"`: BEFORE upgrading, scan the finding's

schema.json

@@ -115,8 +115,9 @@
         },
         "repro_hints": {
           "type": "string",
-          "description": "Instructions for the reproducer agent on how to trigger the bug."
+          "description": "Instructions for triggering the bug, reached-sink evidence channels, and execution telemetry (e.g. build_profile, sanitizers_used, assertions_disabled, ingress_blocked)."
         },
+
         "production_viability": {
           "type": "string",
           "enum": ["VIABLE", "NON_VIABLE", "SAMPLE_OR_TEST", "CONDITIONAL_VIABLE"],