Web-DL mirrors a website with wget, compresses it with archiver, and streams live progress back to your browser over a Socket.IO channel — then hands you a ready-to-download .zip.

• Safe by design — wget is launched with spawn() and an argument array (never a shell string), so a URL can't be interpreted as a command.
• SSRF protection — the server DNS-resolves each host and refuses private, loopback, link-local and cloud-metadata addresses.
• Tunable downloads — set crawl depth, include/exclude file types, a size quota, wait between requests, page requisites, and whether to follow external links.
• Cancel anytime — stop a running job; the wget process is killed and partial files are removed.
• Concurrency control — a configurable cap with a queue keeps the server from being overwhelmed.
• Live progress — real-time progress bar, file count, current file and downloaded size.
• Download history — list, re-download or delete previously generated zips.
• Auto-cleanup — old zips are swept on an interval so disk usage stays bounded.

git clone https://github.com/nooblk-98/Web-DL.git
cd Web-DL
npm install
npm start

Then open http://localhost:3000, paste a URL, tweak the options, and download.

Every download is built from a fixed set of base wget flags:

wget --mirror --convert-links --adjust-extension --page-requisites --no-if-modified-since <url>

User-supplied options (depth, filters, --no-parent, etc.) are layered on top through a strict allowlist — raw flags from the client are never accepted.

The request flows through the server like this:

Browser ──Socket.IO──▶ socket/socket.js ──▶ lib/jobQueue.js ──▶ wget/index.js (spawn wget)
   ▲                                                                    │
   │  live progress, file counts, status                               ▼
   └──────────────────── archiver/index.js ◀──── mirrored site folder (downloads/)
                                │
                                ▼
                    public/sites/<host>.zip  ──▶ served via express.static + /api/history

1. The client submits a URL and options over Socket.IO.
2. lib/urlGuard.js validates the URL and blocks SSRF targets; lib/wgetArgs.js builds a safe argument array.
3. lib/jobQueue.js enforces the concurrency cap; wget/index.js spawns wget and streams progress.
4. archiver/index.js zips the mirrored folder into public/sites/, the temp mirror is removed, and the download link is sent back.
5. lib/cleanup.js periodically deletes zips older than the configured TTL.

Everything is optional and falls back to sensible defaults. Set via environment variables:

Beyond the Socket.IO download channel, a small REST API manages generated zips:

app.js              Express app wiring (routes, views, error handling)
bin/www             Server entry point (HTTP + Socket.IO + cleanup)
routes/             HTTP routes (index, users, history API)
socket/             Socket.IO download orchestration
lib/                Core logic: urlGuard, wgetArgs, jobQueue, activeJobs, sites, cleanup
wget/               wget process spawning + progress parsing
archiver/           Zips a mirrored site folder
config/             Central config, limits and constants
views/              Handlebars templates
public/             Static assets + generated zips (public/sites)
__tests__/          Jest unit tests

Web-DL is built on top of the original Website-downloader by Ahmad Ibrahim — many thanks for the original work this project is based on.