Have you checked that your feature request isn't already filed?

• I read through FAQ and searched through the past issues, none of which addressed my feature request.
• Yes, I have checked that this feature request isn't already filed.

Description & Motivation

Feedback for Arena Platform — OCR support & document-format accessibility

Most respectfully submit the following structured feedback regarding two interrelated usability and evaluation-integrity deficiencies presently affecting the Arena Platform: (1) the absence of an integrated Optical Character Recognition (OCR) capability for image/PDF inputs and (2) the restriction of uploads to only PDF and image formats (no native support for .doc, .docx, .xls, .xlsx, .rtf, .txt, etc.). Together these limitations materially impede fair, efficient and reproducible comparison of model capabilities on the platform.

Executive summary

• Problem: Arena currently requires users to upload content as images or PDF only, and there is no platform-side OCR to extract machine-readable text.
• Consequence: Model evaluations frequently depend on how text is encoded or presented (image vs text). Without OCR and native document support, comparisons are inconsistent, non-reproducible and biased towards models that coincidentally handle image/PDF inputs better.
• Recommendation (high priority): Add (A) robust OCR integration for image/PDF uploads and (B) native ingestion/parsing of common document formats, plus UI elements allowing users to preview, correct and confirm extracted text before benchmarking.

Why this matters — concrete user impacts

1. Evaluation fairness: Two identical prompts supplied as a .docx and as a scanned PDF will be treated differently if Arena cannot normalise content as text. This skews results and undermines confidence in leaderboard comparisons.
2. Usability: Users often possess source files in .docx/.xlsx/.txt form. Forcing conversion to PDF/image is an unnecessary burden and introduces conversion artefacts.
3. Reproducibility & auditability: Extracted text must be visible and downloadable alongside model outputs for verification and legal/regulatory audit. Image-only storage makes this cumbersome.
4. Accessibility: Blind and low-vision users depend on machine-read text. Lack of OCR reduces accessibility compliance.
5. Internationalisation: OCR must support multiple languages and scripts to fairly evaluate multilingual models.

Recommended technical approach (practical, modular)

A. OCR integration (must-have)

• Capabilities: Text extraction from raster images and scanned PDFs, page segmentation, layout retention, multi-language detection, confidence scores per snippet.

• Options:

  • Open-source: Tesseract (with LSTM), enhanced by layout parsers (e.g. layoutparser) for complex documents.
  • Commercial/cloud: Google Cloud Vision, AWS Textract, Azure Cognitive Services — provide higher accuracy, table detection and handwriting recognition.

• Deliverables: Extracted plain text + structured representation (paragraphs, headings, tables), per-token confidence, and a link to the original source.

B. Native document parsing (must-have)

• Supported types: .txt, .md, .docx, .odt, .rtf, .pdf (text+scanned), .xls/.xlsx/.csv.

• Libraries/approaches:

  • .docx: python-docx or Apache POI (Java).
  • Spreadsheets: openpyxl, xlrd or Apache POI. Provide table extraction into CSV/JSON.
  • Generic extraction: pandoc/LibreOffice headless conversion as fallback to normalise formats.

• Feature: Allow users to designate which sheet/column to use for text prompts when uploading spreadsheets.

C. Preprocessing & user confirmation UI (must-have)

• Preview step: After upload, show extracted text side-by-side with the original (rendered PDF/image or document). Allow in-browser edits and an “Accept as ground truth” button.
• OCR settings: Language selector, enable/disable OCR, threshold for low-confidence highlighting, manual correction suggestions.
• Metadata: Record extraction method (engine/version), timestamp, language and confidence; store with evaluation record.

D. Backend & API considerations

• Processing model: Use worker queues for heavier files; publish progress/status in UI. Provide synchronous processing for small files and asynchronous for large ones (with clear UI progress).
• API: Expose endpoints so power users can programmatically upload native documents and retrieve extracted text and annotated outputs.
• Storage: Save both original file and canonical extracted text to ensure reproducibility.

UX / Web UI proposals (language for designers)

• Upload area copy: “Drag or click to upload — supported: PDF (text or scanned), PNG, JPG, DOCX, TXT, XLSX, RTF.”

• Post-upload modal:

  1. Title: “Preview extracted text”
  2. Left pane: original rendering; Right pane: extracted text (editable).
  3. Buttons: [Edit] [Accept as ground truth] [Re-run OCR with language: ⌄]
  4. Show: OCR engine used, extraction confidence, and “Download canonical text” option.

• Benchmark run options: “Use extracted text” vs “Use original file (binary input)”; allow comparisons side-by-side.

Acceptance criteria / success metrics

• Users can upload .docx, .txt, .xlsx, .pdf, .png, .jpg, .rtf.
• For scanned PDFs/images, platform provides extracted text in >90% of simple prose pages (configurable engine and locale permitting), and flags low-confidence regions.
• Extracted text is stored and downloadable alongside model outputs for every evaluation entry.
• Reduction in user complaints about unfair comparisons and fewer manual pre-conversion steps reported within 30 days of deployment (product metric).

Privacy, security & compliance

• Clearly state how uploaded files are stored and who may access them. Provide options: ephemeral processing (no retention) vs retain for reproducibility.
• For sensitive uploads, enable server-side redaction and GDPR/Indian data-protection compliant policies.
• If using third-party OCR cloud services, disclose vendors and offer an on-premises/open-source option for privacy-sensitive organisations.

Phased roadmap (priority, not time)

1. High priority: Add native .docx, .txt and .xlsx ingestion + server-side parsing; implement text preview and “accept ground truth”. Integrate Tesseract for scanned PDFs/images as an initial OCR.
2. Medium priority: Add table detection and spreadsheet cell selection; improve layout retention; add language detection and multi-lang OCR.
3. Low priority: Offer commercial OCR integrations (Google/AWS/Azure) as selectable engines; handwriting recognition; bulk conversion tools and improved QA tooling.

Example brief to include in a developer ticket or support channel

Currently Arena only accepts PDF/image uploads and lacks platform OCR, forcing users to manually convert source files and preventing reliable, reproducible model comparisons. Please add native ingestion for .docx, .txt, .xlsx, .rtf and implement an OCR pipeline (initially Tesseract with an option for cloud OCR). Include a UI preview to confirm extracted text, and store the canonical text with each evaluation record. This will materially improve fairness, accessibility and auditability of benchmark results.

Closing remark

Implementing robust OCR and native document support is not merely a convenience feature; it is essential infrastructure for credible model evaluation. It will materially enhance fairness, user experience and the platform’s standing as a trustworthy benchmarking environment. If helpful, I can draft a succinct changelog entry and image/text UI mockups that Arena’s product and engineering teams can action directly.

Respectfully submitted.

Pitch

No response

Alternatives

No response

Additional context

No response