feat: preserve provenance for local datasets

[aim9sour]

Summary

This adds verifiable native reproducibility for local Hugging Face datasets.save_to_disk() inputs when they are exact materializations of pinned public datasets.

• add a typed provenance block with the public dataset ID, exact 40-character commit SHA, optional dataset configuration, source split, ordered row indices, prompt column, and content SHA-256
• verify both the local materialization and an independently rematerialized public, ungated source before offering reproducibility
• rebuild a missing/sanitized local input from the exact public source during heretic --reproduce
• remove local paths from reproduce.json, config.toml, and the generated README
• emit schema v4 only when materialized provenance is present; continue emitting and reading schema v3 for ordinary Hub inputs

This does not change the one-prompt-per-line TXT contract and does not upload any dataset.

Investigation

I reproduced this on current master at bedb94ef117a271532ac2058447fbc165d5051bd.

The minimal case uses:

• fka/awesome-chatgpt-prompts
• revision ca0bf873b687e093f27beaddce8421f92d8ea7b4
• split train
• rows [3, 104]
• column prompt

Both selected prompts contain embedded newlines. Saving the selected rows with save_to_disk() preserves those records and Heretic loads them successfully through load_from_disk(). However, the local path fails the native reproduction eligibility gate. If reproduction metadata is generated directly, it contains only the machine-local path and cannot rebuild the dataset elsewhere.

I also searched the open and closed issues, PRs, and commit history for local datasets, save_to_disk, load_from_disk, reproducibility, reproduce.json, provenance, materialization, hashes, and pinned datasets. Existing work such as #33 adds local loading but does not preserve reconstructable provenance. The historical local-dataset exclusion was deliberately conservative because incomplete metadata would make a reproduction package misleading; this change preserves that policy for unverified local paths and adds the missing verification.

Provenance and verification

Example:

[good_prompts]
dataset = "/data/materialized-prompts"
split = "train[:]"
column = "prompt"

[good_prompts.provenance]
dataset = "fka/awesome-chatgpt-prompts"
revision = "ca0bf873b687e093f27beaddce8421f92d8ea7b4"
split = "train"
indices = [3, 104]
column = "prompt"
content_sha256 = "b4dab93108fc06d1c03fafbbf8c000274c6777964124bccc24681f42e0f41558"

The content hash is domain-separated SHA-256 over the ordered prompt sequence. Each exact UTF-8 prompt is prefixed with its unsigned 64-bit big-endian byte length, preserving order, record boundaries, empty values, and multiline content without normalization.

A local dataset is eligible only when:

1. the provenance schema is valid and contains an exact commit SHA;
2. the local prompt column matches the declared hash;
3. the source repository is public and ungated;
4. the declared source revision resolves to the same exact commit; and
5. rematerializing the source split and ordered indices produces the same hash.

A mismatch aborts verified loading or disables the reproducibility offer with an actionable reason. Unsupported arbitrary transformation fields are rejected rather than ignored.

Privacy and compatibility

• no raw prompts are included in reproduction files
• no local dataset path is included in reproduction JSON, TOML, or README
• no local dataset is uploaded automatically
• a local path without provenance remains ineligible
• ordinary Hub-only manifests remain schema v3
• heretic --reproduce accepts both v3 and v4
• old Heretic releases reject v4 rather than silently ignoring the new selection semantics

Tests

• ruff format --check .
• ruff check --extend-select I .
• ty check --error-on-warning .
• python -m unittest discover -s tests -p 'test_*.py' — 33 passed, 1 opt-in network test skipped
• HERETIC_RUN_NETWORK_TESTS=1 python -m unittest tests.test_public_dataset_provenance -v — passed
• uv build — sdist and wheel built successfully

The network integration test performs the full data path: fetch exact public revision → select exact multiline rows → save_to_disk() → verified local load → schema-v4 path sanitization → settings reconstruction → public rematerialization → identical prompt list and SHA-256.

[p-e-w]

Hey, thanks for the PR, but this would grow Heretic's entire codebase by about 20%. Which I'm afraid is absolutely not going to happen for a minor feature like this. Not sure what else to say here, this is too much code by a factor of 10 at least 😢