Skip to content

fix: keep severity calibration consistent across scan modes - #737

Open
cjc0013 wants to merge 2 commits into
openai:mainfrom
cjc0013:cjc0013/issue-671
Open

fix: keep severity calibration consistent across scan modes#737
cjc0013 wants to merge 2 commits into
openai:mainfrom
cjc0013:cjc0013/issue-671

Conversation

@cjc0013

@cjc0013 cjc0013 commented Aug 29, 2026

Copy link
Copy Markdown

Summary

Deep reduction could silently assign a different severity from Standard scan findings even when the underlying evidence was materially equivalent. This change preserves equivalent calibration and requires evidence-backed explanations when a reducer changes or reconciles severity.

Closes #671.

Changes

  • compare each Deep reduction severity with every claimed Standard source finding
  • require both severity.rationale and severity.changeConditions when source severities conflict or the output changes their calibration
  • clarify in the reducer prompt that severity is evidence-driven rather than mode-, worker-, repetition-, or wording-driven
  • add regression coverage for silent recalibration, conflicting source severities, and an explicitly justified reconciliation

Validation:

Passed:

  • cmd.exe /d /c pnpm --dir sdk/typescript run build:plugin
  • cmd.exe /d /c node plugins/codex-security/mcp-app/tests/test_artifact_deep_reducer.mjs
  • pnpm --dir sdk/typescript run test:mcp
  • npm --prefix plugins/codex-security/mcp-app run typecheck
  • python -m ruff check --config plugins/codex-security/pyproject.toml .github/scripts/check_plugin_source_compatibility.py .github/scripts/test_check_plugin_source_compatibility.py plugins/codex-security
  • python -m ruff format --check --config plugins/codex-security/pyproject.toml .github/scripts/check_plugin_source_compatibility.py .github/scripts/test_check_plugin_source_compatibility.py plugins/codex-security
  • python .github/scripts/check_plugin_source_compatibility.py

Risk and rollout

The new check applies only after Deep reduction has retained and validated source-finding provenance. Existing reductions that preserve a unanimous source severity are unchanged. A reducer that changes severity or resolves conflicting source severities without the required explanation now fails closed and can be retried with the missing evidence fields.

Prior work and attribution

  • soyeon-oai authored issue Keep severity calibration consistent across scan modes #671, documented the cross-mode calibration inconsistency, and defined the expected evidence-backed behavior.
  • su-jin1425 expressed interest in investigating the issue and explicitly requested confirmation before starting. No patch, diagnosis, design, or code from that comment was used.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.
@github-actions github-actions Bot added the bug Something isn't working label Aug 29, 2026
@cjc0013
cjc0013 marked this pull request as ready for review August 29, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

1 participant