SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning Traces

Source documents are computer science research papers in PDF form, drawn from arXiv. The current release covers 3,058 papers. Papers are processed page by page rather than as a whole, and each page is rendered independently at 150 DPI so the layout detector sees a consistent, fixed-resolution input.

The extraction pipeline applies the following steps to each rendered page:

1. 1.

   Layout detection. A YOLOv8 model fine-tuned on the DocLayNet layout-detection taxonomy finds Picture and Caption regions on the page.

2. 2.

   Precise cropping. Each detected figure region is cropped directly from the source PDF (not the rasterized page image) using PyMuPDF, which keeps the image quality high.

3. 3.

   Caption matching. Each cropped figure is paired with its nearest caption using a heuristic that combines vertical gap and horizontal bounding-box overlap, since captions aren’t always placed directly beneath their figure.

4. 4.

   Context linking. The full body text of the paper is searched for the first sentence that mentions the figure’s number (handling inconsistent styles like "Figure 3," "Fig. 3," and "Fig.3"), and that sentence becomes the figure’s contextual grounding.

5. 5.

   Cleaning and filtering. Records with no detectable caption, or where no figure number could be parsed, are flagged (ref_sentence_found = False) instead of being silently dropped, so downstream users can see exactly how complete the extraction is.

This pipeline produced 29,887 extracted figures across the 3,058 source papers.

Each cleaned (figure, caption, context) triple goes through a question-and-reasoning generation stage with two paths:

• •

  AI-assisted generation (primary path). The Gemini API (across the gemini-2.5-flash, gemini-3.1-lite-flash, and gemini-3.5-flash model versions used during development) generates a question, an answer, and, where possible, a chain-of-thought reasoning trace formatted as <think>...</think><answer>...</answer>, conditioned on the figure image, caption, and referencing sentence.

• •

  Template-based generation (fallback path). When no API key is configured, or generation otherwise fails, a deterministic template system builds a structurally valid question-answer pair from the caption and context alone. This guarantees the pipeline never stalls or produces an empty record just because an external service is unavailable.

Every generated question is assigned one of seven types: component, relationship, process, result, comparison, architecture, or general, so the dataset isn’t dominated by a single style of question.

Each record is stored as a 14-field structured object (identifiers, caption, context, question, answer, question type, generation source, reasoning trace, reasoning source, and the base64-encoded cropped image). It is serialized two ways: a full inspection-ready JSON record, and a lightweight training record already formatted in the target model’s chat-message structure, ready to use without further conversion.

Table 1 lists every field in the record format, its type, and its meaning. This schema is the same across all three dataset versions described in Section 4 (Scaffold-157K, Scaffold-37K, and Scaffold-12K).

Two forms are produced from this schema for every record: a full form (to_dict()) that writes all 14 fields, including the base64 image, for manual inspection and archiving; and a compact training form (to_train_record()) that drops the inspection-only fields and reformats the record directly into the chat-message structure the target vision-language model expects, ready for a training script with no extra conversion step.

To make the schema concrete, Table 2 shows a shortened example record, based on a cross-attention diagram figure.

Across the full release, these question-answer pairs are drawn from 29,887 figures extracted from 3,058 papers. The larger Scaffold-157K and Scaffold-37K collections show that the pipeline can extract data at scale. The smaller Scaffold-12K collection, drawn from a small number of fully processed papers, was manually spot-checked for quality and used for the baseline training run reported in Section 5.

Every generated question is labeled with one of seven types (Section 3.3). Table 4 shows the target distribution the generation pipeline aims for across a balanced extraction run; relationship and component questions are intentionally favored, since these best test whether a model has actually understood how a diagram’s parts connect, rather than just what appears in it.

Each record also carries provenance metadata (qa_source and cot_source) showing whether its question/answer and reasoning trace came from the Gemini API path or the synthetic template fallback. Table 5 summarizes what each source path guarantees.

Users who want a purely AI-supervised question set can filter to qa_source = "gemini" records. Users who want a guaranteed-available, simpler question set across the whole corpus can rely on the synthetic fallback records, which are always structurally valid even though their phrasing is more templated.

Following standard dataset documentation practice, we summarize the dataset below.

Motivation. The dataset closes a gap: there was no figure-question-reasoning corpus for computer science architecture diagrams, flowcharts, and pipeline schematics, which is needed to train domain-specific vision-language models.

Composition. Each instance is a single figure from a computer science paper, paired with its caption, its referencing sentence from the paper body, a generated question, a generated answer, a question-type label, a chain-of-thought reasoning trace (where available), and provenance metadata showing whether the question/answer and reasoning came from AI generation or the template fallback.

Collection Process. Instances are generated automatically from arXiv PDF source files using the pipeline in Section 3; no individual figure was manually annotated, though a sample of 1,000 was manually reviewed for quality.

Preprocessing/Cleaning. Figures without a detectable caption or a parsable figure number are flagged rather than dropped outright, and records missing a referencing sentence are marked accordingly so downstream users can filter as needed.

Uses. The dataset is meant for fine-tuning and evaluating vision-language models on computer science diagram understanding, including chain-of-thought reasoning generation. It is not intended for non-CS scientific figures or general-purpose visual question answering.

Distribution. See Section 7, Data Availability Statement.

Maintenance. The pipeline that produced this dataset is meant to be re-run on additional papers as the project grows; the dataset is expected to keep growing rather than stay static.

• •

  Scale of verification. Only the Scaffold-12K split has been manually spot-checked; the larger Scaffold-157K and Scaffold-37K collections have not been independently checked for record quality at the same level of scrutiny.

• •

  Domain coverage. Source papers so far come from a limited number of computer science subfields; diagram styles from less-represented subfields may be underrepresented.

• •

  Generation dependency. A large part of question, answer, and reasoning quality depends on the Gemini API; the template-based fallback, while reliable, produces simpler and more uniform questions than the AI-assisted path.

• •

  Layout detector generality. The figure/caption detector is trained on a general-purpose document layout taxonomy (DocLayNet) rather than one fine-tuned specifically for computer science architecture diagrams, so detection accuracy on unusual layouts (e.g., multi-panel figures, diagrams embedded in tables) hasn’t been formally measured.

• •

  Reasoning trace reliability. Chain-of-thought traces are model-generated (or template-generated) rather than human-written, and haven’t gone through a formal human evaluation of factual correctness or reasoning quality at scale.

Copyright and source material. All source papers come from arXiv, a repository of preprints made publicly available by their authors. Extracted figures and captions are derivative excerpts used for research purposes; redistributing the dataset should respect the license each source paper was posted under (arXiv papers carry a range of licenses, including some non-permissive ones), and anyone redistributing cropped figures at scale should verify per-paper licensing rather than assume uniform terms.

Attribution. Each record keeps a paper_id field linking back to its source paper, preserving traceability and enabling attribution to the original authors.

Use of a third-party generation API. Part of the dataset’s questions, answers, and reasoning traces are generated by a third-party AI service (Gemini). This raises two considerations: (1) AI-generated fields may contain factual errors or hallucinated reasoning, which downstream users should account for, especially before using the dataset to train models for high-stakes use, and (2) figure images, captions, and text excerpts are exposed to a third-party API during dataset construction, which should be disclosed to any parties whose papers are processed under stricter data-handling requirements.

No personal or sensitive data. The dataset consists of technical diagrams and their captions from computer science papers; it is not expected to contain personal, biometric, or otherwise sensitive information about individuals, though authors’ names may appear incidentally within paper text used as context.