Skip to content

Commit 6247efb

Browse files
committed
Refine risk calibration rules in mantis_calibrate skill
Updated `mantis_calibrate/SKILL.md` to improve risk calibration accuracy and reduce false positives (over-estimations) based on empirical analysis of historical ratings and reviewer feedback. Key changes: - Capped internal component threats (exposure < 1.0) at HIGH priority, with an exception for core in-cluster infrastructure (CSI/CNI) escaping to host/cross-tenant (which remains CRITICAL eligible). - Capped local attack vectors (e.g., SUID LPE) at MEDIUM priority. - Capped rarely exposed or unlikely user-controlled threats at MEDIUM priority. - Enforced stored XSS on critical admin pages to cap at HIGH (preventing CRITICAL XSS). - Added a rule to elevate impact for core security control bypasses (e.g., auth bypass) to avoid under-estimation. - Reorganized Section 3 (Sanity Triage) into explicit Force-Downgrade and Force-Cap sections. - Set exposure default to 0.8 (Internal) when threat model is missing. - Added rules to handle probabilistic LLM vectors (capping at HIGH/MEDIUM, with retry exception). - Added rules to cap non-default configurations at HIGH (default) or MEDIUM (if documented insecure). - Added rules to cap equivalent primitives at MEDIUM (default) or force-downgrade to LOW (if attacker already has shell access of same/higher privilege). - Added rules to cap vulnerabilities requiring supply-chain or build-time prerequisites at HIGH. - Explicitly documented that Sanity Triage overrides base scoring upgrades. Tested with an 18-case validation suite covering all these edge cases. TAG=agy CONV=258932e0-42ce-426c-8f4c-508a3d272c26 Change-Id: Ib9a8abcc498ec35b87e7382b55d68e563cfcc247
1 parent e0e9c92 commit 6247efb

1 file changed

Lines changed: 121 additions & 38 deletions

File tree

‎mantis_calibrate/SKILL.md‎

Lines changed: 121 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,13 @@ Execute the calibration as follows:
6868
the type "the code is fragile", "lack of defense-in-depth", or
6969
purely theoretical hygiene issues MUST have an Impact score of 1,
7070
ensuring they are rated LOW at most.
71+
- **Security Control Bypass (Upgrading):** If the vulnerability
72+
directly bypasses a core security control (e.g., authentication,
73+
authorization, cryptographic signature verification) or defeats the
74+
primary security purpose of a library (e.g., a library meant to
75+
secure keysets allows attacker control), elevate the Impact score to
76+
at least **4** (or **5** if it leads to systemic compromise), even
77+
if the immediate technical impact seems localized.
7178
- *Note on Privileges Required & Lateral Movement:*
7279
- If the finding requires **HIGH** privileges (e.g.,
7380
administrative privileges, admin-to-super-admin escalation) or
@@ -77,6 +84,7 @@ Execute the calibration as follows:
7784
- If the finding requires **LOW** privileges (e.g., standard
7885
authenticated user), cap its individual Impact score at **3**
7986
(unless it leads to systemic compromise of other tenants/users,
87+
OR it directly bypasses a core security control/library purpose,
8088
in which case it can be higher).
8189
- These caps apply to *individual* findings. If successfully
8290
chained into an Exploit Chain (Super Finding) by the chainer,
@@ -105,6 +113,18 @@ Execute the calibration as follows:
105113
- If it resides in an **Internal Component** accepting
106114
semi-trusted parsed data: 0.8.
107115
- If deeply nested inside a **Privileged/Trusted Zone**: 0.5.
116+
- **Inference when Threat Model is Missing/Incomplete:** If
117+
`workspace/kb/THREAT_MODEL.md` does not exist or does not
118+
mention the component:
119+
- Analyze the file path, imports, and caller hierarchy to
120+
infer exposure (e.g., public APIs vs internal helpers).
121+
- Default the Exposure Multiplier to **0.8** (Internal)
122+
unless there is clear evidence of direct external
123+
exposure.
124+
- If the finding description, history, or critic reasoning
125+
suggests the component is "rarely exposed", "internal
126+
only", or "unlikely to be attacker-reachable", reduce
127+
the Exposure Multiplier to **0.5** or lower.
108128
- **Asset Criticality & Reachability:**
109129
- If the Threat Model indicates the component handles
110130
high-value data (e.g., PII, core secrets), keep the
@@ -113,7 +133,8 @@ Execute the calibration as follows:
113133
sandboxed test data), reduce the multiplier (e.g., 0.5).
114134
- **Availability-Specific Context:** If the finding is
115135
availability-only (DoS), check the component's
116-
`availability_tier` in the Threat Model:
136+
`availability_tier` in the Threat Model (if missing, default
137+
to STANDARD):
117138
- `LOW_CRITICALITY`: Reduce multiplier to **0.5**.
118139
- `STANDARD`: Reduce multiplier to **0.8**.
119140
- `CRITICAL`: Keep multiplier at **1.0**.
@@ -128,8 +149,6 @@ Execute the calibration as follows:
128149
required, the combined multiplier is 0.8 * 0.7 = 0.56). This
129150
ensures these findings are capped below the CRITICAL
130151
threshold.
131-
- If `workspace/kb/THREAT_MODEL.md` does not exist or does not
132-
mention the components, default the Multiplier to 1.0.
133152
- If `production_viability` is **SAMPLE_OR_TEST**:
134153
- Set the Context Multiplier to a reduced value (e.g. `0.4`) so
135154
that severe bugs in sample code typically land in the MEDIUM
@@ -147,41 +166,105 @@ Execute the calibration as follows:
147166
damage, user sentiment fallout) is taken into account. Do *not* include the
148167
outrage factor in the final numerical score.
149168

150-
3. **Critical Sanity Triage (Downgrading Weak Findings):** Before determining
151-
the final priority, perform a second-level sanity check on the quality of
152-
the finding and its accumulated evidence. You **MUST** force-downgrade the
153-
finding's priority to **LOW** (and cap its final score at **2.0**) if it
154-
meets any of the following "weak finding" criteria:
155-
156-
- **Reproduction Failure or Not Attempted:** The reproduction failed
157-
(`repro_status: "failed_to_reproduce"`) or was not attempted
158-
(`repro_status: "not_attempted"`). If the agent did not successfully
159-
reproduce the vulnerability, it MUST be force-downgraded to **LOW**
160-
priority (and cap its final score at **2.0**) regardless of any
161-
theoretical arguments for production viability.
162-
- **Static Confirmation:** The vulnerability was statically confirmed but
163-
not empirically reproduced (`repro_status: "statically_confirmed"`). To
164-
account for the lack of empirical proof:
165-
- Force-cap the `likelihood_score` at **3** (or keep it lower if
166-
appropriate).
167-
- Apply an additional **0.8** multiplier to the final calculated score
168-
(Hazard).
169-
- The finding **MUST NOT** be classified as **CRITICAL** priority (if
170-
the score lands in the CRITICAL range, downgrade the priority to
171-
**HIGH**).
172-
- **Minor Configuration Hygiene:** The issue represents a minor deviation
173-
from best-practice configuration (e.g., slightly loose permissions on an
174-
internal directory, lack of modern encryption on low-value internal
175-
transport) but does not lead to a clear exploit path, privilege
176-
escalation, or data exposure.
177-
- **Vague Code Paths / Fragile Assumptions:** The finding's description or
178-
reasoning relies on unverified assumptions about caller behavior or
179-
adjacent system components that are not documented in the active
180-
Knowledge Base.
181-
- **Unreliable/Noisy Triggers:** The finding represents an issue that can
182-
technically be triggered but is highly likely to be ignored in practice
183-
due to high noise, or is indistinguishable from normal system operations
184-
without causing real harm.
169+
3. **Critical Sanity Triage (Downgrading & Capping Findings):** Before
170+
determining the final priority, perform a second-level sanity check on the
171+
quality of the finding, its context, and accumulated evidence. Sanity Triage
172+
caps and downgrades override any upgrades calculated in Section 2 (including
173+
the Security Control Bypass upgrade). You **MUST** force-downgrade or cap
174+
the finding's priority and score if it meets any of the following criteria:
175+
176+
* **Force-Downgrade to LOW (Cap at 2.0 / LOW Priority):**
177+
178+
- **Reproduction Failure or Not Attempted:** The reproduction failed
179+
(`repro_status: "failed_to_reproduce"`) or was not attempted
180+
(`repro_status: "not_attempted"`). Regardless of theoretical
181+
production viability.
182+
- **Unreachable / Uncontrolled Inputs:** The finding relies on inputs
183+
that are documented as highly unlikely to be user-controlled, and no
184+
path from a trust boundary is proven.
185+
- **Third-Party / Supply Chain Reachability:** Vulnerabilities in
186+
third-party libraries (dependency CVEs) where a reachable path from
187+
application input to the vulnerable function has not been actively
188+
demonstrated.
189+
- **Minor Configuration Hygiene:** Minor deviations from best practice
190+
(e.g., slightly loose permissions on internal dirs, lack of modern
191+
encryption on low-value internal transport) without a clear exploit
192+
path.
193+
- **Non-Security Critical Components:** The finding affects a
194+
component or data with no security sensitivity (e.g., public info,
195+
signatures on non-security payloads, cosmetic outputs).
196+
- **Vague Code Paths / Fragile Assumptions:** Relying on unverified
197+
assumptions about caller behavior or adjacent system components.
198+
- **Unreliable/Noisy Triggers:** Triggers that are likely to be
199+
ignored in practice or indistinguishable from normal operations.
200+
- **Prerequisite Shell Access (Equivalent Primitives):** The attacker
201+
already possesses local shell access on the target container or host
202+
with the **same or higher** privilege level than the exploit
203+
provides, rendering the gained access redundant (e.g., exploiting a
204+
bug to get a standard user shell when already logged in as a
205+
standard user, or exploiting a local buffer overflow to run commands
206+
as root when already running as root). This does NOT apply to
207+
low-to-high privilege escalation (e.g., standard user to root),
208+
which should cap at MEDIUM.
209+
210+
* **Force-Cap to HIGH (Cap at 7.9 / Maximum HIGH Priority):**
211+
212+
- **Static Confirmation:** Statically confirmed but not empirically
213+
reproduced (`repro_status: "statically_confirmed"`). Cap
214+
`likelihood_score` at **3**, apply **0.8** multiplier to Hazard, and
215+
MUST NOT be CRITICAL.
216+
- **Strict XSS Caps:** All XSS vulnerabilities. Default to MEDIUM or
217+
LOW; cap at HIGH (7.9) only for stored XSS on critical admin pages
218+
with zero-click execution for the admin.
219+
- **Internal / Nested Components:** Any finding with a Network/Trust
220+
Exposure multiplier less than 1.0 (i.e., Internal Component or
221+
Privileged Zone). If the calculated score lands in the CRITICAL
222+
range, downgrade the priority to HIGH. *Exception:* Do NOT cap at
223+
HIGH if the component is core in-cluster infrastructure (e.g., CNI,
224+
CSI, admission webhook, service mesh) AND the impact escapes to the
225+
host node (e.g., node-root file R/W) or allows cross-tenant
226+
escalation. These remain eligible for CRITICAL.
227+
- **Probabilistic LLM Vectors:** Attacks relying on probabilistic LLM
228+
behavior (e.g., prompt injection, jailbreaking) to trigger a
229+
vulnerability. Cap at **HIGH** (7.9) and default to **MEDIUM** or
230+
**LOW**. *Exception:* If the attacker can query the LLM/system
231+
repeatedly without rate limits, concurrency limits, or security
232+
blocking/alerting that would impede the attack (allowing them to
233+
brute-force and effectively eliminate the non-determinism), this cap
234+
may be lifted.
235+
- **Supply-Chain / Build-Time Prerequisites:** If the exploit requires
236+
the attacker to already possess a supply-chain position (e.g.,
237+
ability to poison dependencies, modify upstream source) or write
238+
access to the build pipeline to trigger the vulnerability. Cap at
239+
**HIGH (7.9)** since the entry barrier is extremely high, but the
240+
downstream compromise is systemic. (Force-downgrade to LOW/2.0 only
241+
if they already possess shell access on the target, as per the
242+
Prerequisite Shell Access rule).
243+
- **Non-Default Configurations:** Findings that are only exploitable
244+
under non-default configurations. Cap at **HIGH (7.9)** to reflect
245+
the additional configuration barrier.
246+
247+
* **Force-Cap to MEDIUM (Cap at 5.9 / Maximum MEDIUM Priority):**
248+
249+
- **Local Attack Vector:** Vulnerabilities requiring local shell
250+
access (e.g., local privilege escalation, SUID exploitation) without
251+
VM escape. (Downgrade to LOW/2.0 if it only affects a single user's
252+
isolated data).
253+
- **Intra-Customer / Same-Tenant:** Attacks restricted to the same
254+
tenant boundary the attacker already controls, with no cross-tenant
255+
escalation or host compromise.
256+
- **Rarely Exposed Components:** Findings in components documented as
257+
'rarely exposed' or 'unlikely to be user controlled'.
258+
- **Equivalent Primitives (No Boundary Breach):** The attacker profile
259+
capable of triggering the vulnerability already possesses equivalent
260+
access, privileges, or capabilities (primitives) through standard
261+
system features (e.g., an admin exploiting a bug to download a file
262+
they can already download via the UI). Cap at **MEDIUM (5.9)** to
263+
maintain visibility for defense-in-depth cleanup.
264+
- **Documented Insecure Configurations:** Non-default configurations
265+
that are explicitly documented in public manuals as insecure,
266+
diagnostic-only, or strictly non-production. Cap at **MEDIUM
267+
(5.9)**.
185268

186269
4. **Determine Priority:**
187270

0 commit comments

Comments
 (0)