Turn a web crawl into an ML-ready dataset. Extract, filter, deduplicate, embed, shard.
You scraped 50,000 pages. You cannot train on them, because a scraped page is not a document.
In: raw HTML, one JSON object per line.
{"url": "https://example.com/gpu-guide",
"html": "<html><head><title>GPU scheduling on Kubernetes</title></head><body><nav><a href='/'>Home</a><a href='/about'>About</a></nav><article><h1>GPU scheduling</h1><p>Kubernetes exposes GPUs through the NVIDIA device plugin, which advertises nvidia.com/gpu as an allocatable resource on every node...</p></article><footer>© 2026 Example Inc. Privacy Policy</footer></body></html>"}Out: clean text, deduplicated, quality-scored, in gzipped shards a dataloader can stream.
{"doc_id": "dc1bf1ffa1945c31",
"url": "https://example.com/gpu-guide",
"title": "GPU scheduling on Kubernetes",
"text": "GPU scheduling\n\nKubernetes exposes GPUs through the NVIDIA device plugin, which advertises...",
"quality": {"passed": true, "rules": {"word_count": {"passed": true, "value": 104, "threshold": 50}}},
"signatures": {"raw": "81879111...", "normalized": "0745f9c2...", "structural": "a74bccb9..."}}The nav, the footer, and the copyright line are gone. The title came from <title>, the body from
the article, and nothing else survived. Across a real crawl, so do the 9,884 pages that were the
same article under a different URL and the 14,022 that were link farms, stub pages, or SEO spam.
Both blocks above are actual output, not illustration. docs/quickstart.md reproduces them in
about a minute.
That gap is the whole product. Crawling is solved; Scrapy does it. Training is solved; your framework does it. The part in between, where a crawl becomes a corpus, is where people quietly lose weeks and then train on duplicates anyway.
A crawl becomes a corpus in one process, on any machine, with every decision attributable.
- You can always answer "why is this document not in my dataset?" Every drop
is attributed to a rule, every rule reports the value it observed and the
threshold it failed, and the counts reconcile:
seen == kept + dropped + malformed. A filter you cannot attribute is a filter you cannot tune. - It runs where the work is. The core imports stdlib only, so it works inside someone else's container, on a locked-down build box, or air-gapped. Permanent, and enforced by CI.
- It stops where it should. Below roughly ten million documents, one process
with no dependencies is the right trade. Above that it is the wrong one, and
the honest answer is HuggingFace's
datatrove.
Full detail, plus what is deliberately not planned, in the roadmap. Work in flight is tracked in the 0.2 and 0.3 milestones.
- You are building an LLM or RAG corpus from crawled pages and need dedup that catches near duplicates, not just byte-identical ones.
- You run a scraper in production and want the cleaning stage to be a library with tests rather
than a 400-line
clean.pynobody wants to touch. - You cannot install dependencies where the cleaning has to run: someone else's container, a locked-down build box, an air-gapped environment.
If you want a distributed, Spark-scale pipeline with a cluster behind it, use HuggingFace's
datatrove. websieve deliberately runs in one process with no dependencies. Its
reproducible benchmark found that memory is the binding constraint: peak RSS
grew linearly by about 10.8 MiB per 1,000 input documents because the exact and LSH indexes retain
keys and signatures for every kept document. On that controlled corpus, the fitted RSS reaches 16
GiB at roughly 1.5 million inputs—not the previously claimed ten million. Corpus shape and settings
move that wall, so rerun the benchmark on representative input before sizing a production job.
crawl -> extract -> normalize -> exact dedup -> quality -> near dedup -> embed -> shards
boilerplate NFKC 3 levels 9 rules MinHash+LSH batched
Each stage costs more than the one before it, so each shrinks the input to the next. One hash beats
nine heuristics; nine heuristics beat 128 hash permutations. Details in
docs/architecture.md.
The core has no dependencies. Not "few". None. pyarrow, torch, and scrapy are optional
extras used only by the stages that genuinely need them, and CI fails if a third-party import ever
reaches the core.
The committed benchmark uses a checksum-pinned Common Crawl WET record and a deterministic 1,000-document corpus with known near-duplicate families. On macOS arm64 with CPython 3.12, the 128-permutation configuration processed 351 documents/s with precision and recall of 1.0. Dropping to 64 permutations reached 689 documents/s but recall fell to 0.98 (4 of 200 duplicates missed).
Peak RSS was 42 MiB at 1,000 input documents, 86 MiB at 5,000, and 140 MiB at 10,000; the fitted
growth is 10.8 MiB per 1,000 inputs. If the process could consume all installed RAM, that implies
upper bounds of about 1.5 million documents at 16 GiB, 3.0 million at 32 GiB, and 6.1 million at 64
GiB. Real limits are lower because the operating system needs headroom. These are one-machine,
controlled-corpus projections—not claims about every crawl. docs/benchmarks.md
records the exact input hash, hardware, commands, extrapolation table and limits, plus the
machine-readable JSON.
pip install websieve # core, zero dependencies
pip install "websieve[parquet]" # + Parquet output
pip install "websieve[embed]" # + GPU embedding
pip install "websieve[all]"Pipe a crawl straight in:
scrapy crawl myspider -o - -t jsonlines | websieve build - -o dataset/Or run against a file:
websieve build crawl.jsonl -o dataset/ --threshold 0.85 --shard-size 50000You get sharded output, a manifest, and a stats report:
While build runs, it reports live counters and throughput to stderr every
10,000 input records, including malformed records. This keeps piped stdout
clean and does not need to know the stream's total length. Change the interval
with --progress-every N, or use --progress-every 0 to disable it.
50000 seen 18342 kept (36.7%) 1240 docs/s
dataset/
shard-00000.jsonl.gz
shard-00001.jsonl.gz
manifest.json every shard, its record count, its size
stats.json what was dropped, and by which rule
seen 50000
kept 18342 (36.7%)
dropped 31658
empty 412
exact duplicate 9884
quality 14022
near duplicate 7340
quality rule failures (a document can fail several):
word_count 8110
terminal_punctuation_ratio 6033
repetition_ratio 2914
That breakdown is the point. A filter you cannot attribute is a filter you cannot tune.
websieve assess crawl.jsonl -v # what would be dropped, and why. Drops nothing.
websieve assess crawl.jsonl --sample 1000 # same, on a sample, for large crawls
websieve dedup crawl.jsonl # duplicate clusters with similarity scores
websieve extract page.html # main-content text from one pagefrom websieve.pipeline import Pipeline, PipelineConfig
from websieve.models import Document
pipeline = Pipeline(PipelineConfig(near_dup_threshold=0.85))
for doc in pipeline.process(Document(url=u, html=h) for u, h in crawl()):
index(doc.text)
print(pipeline.stats.render())Every stage also stands alone:
from websieve.quality.heuristics import assess
from websieve.dedup.minhash import MinHash, LSHIndex
from websieve.clean.boilerplate import extractText-density heuristic, not a readability port. Blocks are scored by length and link density, then the best contiguous run is kept, so navigation and footers fall away because they are short and mostly links. Headings adjacent to the body are reclaimed, because Kadane's algorithm will not pick them up on its own and an article without its title is a worse document.
No dependencies. If you can afford one and need higher accuracy, use trafilatura and feed its
output into the quality stage instead.
Nine rules from the published recipes for large web corpora, chiefly the Gopher rules (Rae et al., 2021) and the C4 cleanup (Raffel et al., 2020): word count, mean word length, symbol ratio, alphabetic ratio, bullet and ellipsis line ratios, terminal punctuation, line repetition, and boilerplate markers.
Reimplemented rather than vendored so every threshold is visible and adjustable. The right values genuinely differ between general web text and a domain corpus, and you cannot tune what you cannot see. Every rule runs even after one fails, so the failure histogram is complete.
The rule set adapts to the writing system. Every one of those rules was derived from English and assumes words are separated by spaces, so applied unchanged they reject ordinary Chinese, Japanese, Thai, and Korean outright. Script is detected per document, and the rules are composed from its traits: not space-delimited means character-based length rules, no terminal punctuation means that rule is dropped, dense characters mean a lower word-length floor. Verified against twelve writing systems.
Two passes, cheap before expensive.
Exact, at three levels. raw is byte-identical. normalized ignores case, punctuation, and
spacing. structural also collapses digits, which catches templated pages differing only by a
price, date, or id. That last one will merge genuinely different pages whose only distinguishing
content is numeric, so choose it deliberately.
Near, by MinHash with LSH banding. Shingle into word n-grams, keep the minimum under num_perm
hash permutations, then band the signature so candidate lookup is a hash hit instead of an O(n^2)
scan. Candidates are verified against real signature similarity afterwards, because LSH returns
candidates, not answers.
Permutations are simulated as (a*h + b) mod p over a Mersenne prime, the standard universal
hashing construction, which is why this needs no numpy. Accuracy is what the theory predicts:
| Pair | True Jaccard | MinHash estimate, 256 perms |
|---|---|---|
| One word differs | 0.833 | 0.863 |
| Unrelated documents | 0.000 | 0.000 |
| Identical | 1.000 | 1.000 |
The model call sits behind a Protocol, so the part that actually governs throughput is testable
without a GPU and CI exercises it with a stub.
Two things dominate embedding throughput, and neither is the model:
- Padding waste. A batch is as slow as its longest sequence. Sorting by length before batching keeps short documents from being padded up to the longest one in the corpus.
- Batch size versus memory.
adaptive_batchescaps onmax_len * batch_sizerather than record count, so a batch of long documents automatically becomes a smaller batch.
padding_efficiency() reports how much of what you processed was real rather than padding. Below
about 0.7, the length distribution is too spread out for the current batch size.
Worth knowing: sorting helps when batch size is the binding constraint. When the token cap binds first, both orderings produce identical batches and sorting buys nothing. Both regimes are asserted in the tests rather than assumed.
Sharded JSONL (gzip) or Parquet. Many medium shards rather than one large file, because shards parallelize across dataloader workers, resume cleanly after a failure, and stream from object storage without a full download.
Both writers emit manifest.json. read_shards() reads it rather than globbing the directory, so
a truncated or partially uploaded dataset raises an error instead of silently yielding fewer records
than you think you have.
| Symptom | Change |
|---|---|
| Keeping too much junk | Lower --threshold toward 0.7; raise the word_count minimum |
| Dropping good documents | Raise --threshold; check stats.json for the dominant rule |
| Dedup too slow | Lower --num-perm to 64 |
| Missing obvious duplicates | Raise --bands for higher recall and more candidates to verify |
| Templated pages surviving | --exact-level structural |
| Low padding efficiency | Lower max_batch_tokens, keep sort_by_length=True |
PipelineConfig exposes all of it. See docs/tuning.md and
docs/architecture.md.
pip install -e ".[dev]"
pytest112 tests, 92 percent line coverage, no network access and no GPU required. The uncovered remainder
is ParquetShardWriter and SentenceTransformerEncoder, which need pyarrow and torch and are
not installed in CI. They are thin call-throughs; the logic they sit behind is covered.
CI runs on Python 3.10, 3.11, and 3.12, and includes a job that fails if the core ever acquires a runtime dependency.
| Guide | For |
|---|---|
| Quickstart | Five minutes, no crawler needed. Real input and real output |
| Architecture | Why the stages are ordered this way, memory behavior, limitations |
| Benchmarks | Reproduce throughput, MinHash accuracy, and peak-memory measurements |
| Tuning | Calibrating thresholds against your own corpus |
| Extending | Swapping the extractor, model, writer, or similarity metric |
| Roadmap | The vision, what is next, and what is deliberately not planned. Mapped to milestones |
| Changelog | What changed in each release |
| Contributing | Setup, the rules specific to this codebase |
Issues and PRs welcome. If an issue is not the right shape, or for commercial use and private corpora, email contact@eprecisio.com.
Maintained by Ehtisham Mubarik (LinkedIn) at Eprecisio Technologies.
This repository began as a containerized Scrapy deployment (Scrapy, Scrapyd, Postgres, Filebeat,
Jenkins) built for a Swiss real-estate crawl. That crawler is still here under
examples/immo_crawl/ as a working integration example — a complete
Scrapy/Scrapyd/Postgres deployment, with its own dependencies and its own README, kept out of the
root so it cannot be mistaken for this package's build configuration. websieve is the part that was missing: everything between a finished
crawl and a dataset you would actually train on.
MIT. See LICENSE.
If websieve saved you an afternoon, a star helps the next person find it.
Contributions welcome: good first issue
is pre-scoped, and CONTRIBUTING.md says how to claim one so
two people do not build it twice.