Honor undecodable-font OCR routing in per-page extraction - #431
Open
yzxcj797 wants to merge 1 commit into
Open
Conversation
extract_pages_markdown ignored the font-resource gate that classify_pdf/detect_pdf_type use to flag pages whose used fonts cannot map back to Unicode (Identity-H/V or custom-encoded Type3, both without ToUnicode). A custom Type3 Encoding can map every byte to a plausible ASCII letter - e.g. a Caesar-shifted alphabet - so the extracted markdown passes the cipher/CID/encoding heuristics and the per-page result returned needs_ocr=false with the wrong native text while detection correctly routed the same page to OCR with suspected_garbled_text. Route the same Phase-3 signal through the shared page_ocr_signals analysis pass (no extra page scan), add the matching OCR reason, and suppress the untrusted markdown. The fixture is the deterministic stdlib-only reproduction from firecrawl#428 (fictional text, no third-party content). Fixes firecrawl#428
![cubic-dev-ai[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Fixes extract_pages_markdown to honor the detector's undecodable-font OCR gate (Type3 without ToUnicode), so extraction and detection agree on needs_ocr; adds a regression test with a deterministic fixture.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Detection correctly routes a page to OCR when its used fonts cannot map back to Unicode, including custom-encoded Type3 fonts without ToUnicode.
extract_pages_markdowndid not consume that font-resource signal. A custom Type3 encoding can map every byte to plausible ASCII letters, so its wrong output could pass output-text heuristics and be returned withneeds_ocr=false.This change routes the existing detection font gate through the shared
page_ocr_signalsanalysis pass:suspected_garbled_text;Fixes #428.
Testing
main: detection flagged the page while per-page extraction returnedneeds_ocr=false.suspected_garbled_text.cargo test— 987 unit, 3 CLI, 163 integration, and 2 documentation tests passcargo fmt --all -- --check— passescargo clippy— no new warnings; existing pristine-main findings are unchangedgit diff --check— passesSummary by cubic
Per-page extraction now honors the detector’s undecodable-font OCR gate so extraction and detection agree. Previously, pages using Identity-H/V or custom-encoded Type3 fonts without ToUnicode could return trusted markdown with needs_ocr=false; now they return needs_ocr with reason suspected_garbled_text and suppress markdown.
Review notes
page_ocr_signals; its return now includes (template image, undecodable fonts, vector text).suspected_garbled_textand suppresses native markdown when undecodable fonts are used.Written for commit 15e07c1. Summary will update on new commits.