FreeCrawl SEO Tool

Open-source desktop SEO crawler — a free, cross-platform alternative to Screaming Frog

🌐 Website · 📦 Download · 🐛 Report Bug · 📝 Changelog

A high-performance website crawler built for serious SEO audits. Targets 1M+ URLs on a single machine, with a dense Screaming Frog–style UI, 150+ SEO issue checks, and zero native dependencies.

FreeCrawl is a desktop SEO crawler that scales to 1M+ URLs on a single machine at 80–150 URL/s via undici with keep-alive, with optional JavaScript rendering through headless Chromium (Playwright) that captures the post-JS DOM, full-page / above-fold / mobile screenshots, LCP candidate elements, and a per-URL mobile usability audit. It runs 150+ on-page SEO checks across 30 top-level tabs — exact and near-duplicate clustering (SimHash + LSH) with a dedicated Cluster view, full hreflang validation (reciprocity, self-reference, inconsistent lang), OpenGraph / Twitter Card / JSON-LD / Web App Manifest parsing, AMP smoke validation, structured-data validation across 17 schema types (duplicate @id, malformed @type, missing required + Google-recommended props), a configurable performance budget (response-time / page-size / LCP / CLS ceilings), WCAG accessibility checks (<main> landmark, skip-link, ARIA roles, heading order), security headers + SSL/TLS chain audit + active/passive mixed content split, readability scores (Flesch, Flesch–Kincaid, Gunning Fog), and a 10-rule custom extraction engine — CSS selectors, XPath, regex, and JSONPath (for JSON API responses) — with a live Preview dialog and JSON import / export for sharing rule sets.

The dense dark UI renders virtualized 1M-row tables that live-stream rows during a crawl, with List ↔ Tree ↔ Cluster view toggles, column pin (sticky-left) + drag-to-reorder + show/hide, advanced AND/OR filters across 24 fields × 12 operators, per-tab quick-filter dropdowns, a 16-tab Details panel per URL (URL Details, Outline, Inlinks, Outlinks, Images, Resources, Extracted Data, SERP Snippet, HTTP Headers, Cookies, Structured Data, View Source, View Rendered, Screenshot, Duplicates, Analytics), an English + Turkish UI including every Settings panel, in-app scheduled crawls (hourly / daily / weekly / custom), a live memory monitor, robots.txt syntax validator, URL rewriting with live preview, project-vs-project compare diff, a Cytoscape link graph in its own native window (force-directed, BFS / radial / Sugiyama tree, and directory-tree layouts, a By-LCP above-the-fold colour overlay, and a click-to-trace Crawl Path Report that highlights the shortest discovery path from the homepage to any page), a standalone Log File Analyzer window (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, and crawl × log orphan detection — with 40+ bot detection, optional reverse-DNS verification, per-bot URL filtering, and one-click CSV / Excel export of every table), and a .seoproject file association for OS double-click.

Integrations cover Google Search Console (clicks, impressions, CTR, position + URL Inspection coverage), Google Analytics 4 (sessions, users, bounce, engagement), PageSpeed Insights Lighthouse audits, custom AI prompts via OpenAI / Anthropic Claude / local Ollama with {url}/{title}/{description}/{h1}/{body} variables, and SEO authority providers (Ahrefs / Majestic / Moz / Semrush) behind a single dropdown — each integration card now ships with an in-app step-by-step Setup Guide modal in English and Turkish that walks you through OAuth client creation, test-user setup, API enablement, and the most common errors. Exports go through a unified Export Crawl Data dialog (Excel .xlsx / CSV UTF-8 / JSON / XML with hierarchical category selection and nested folder output) plus a standalone HTML audit report, sitemap generator (standard / image / hreflang / sharded / gzipped), and direct streaming to Google Sheets and BigQuery. Project files can be saved as password-protected encrypted snapshots (.seoproject.enc, AES-256-GCM + PBKDF2). The MCP server exposes 90 tools that let Claude Code or any MCP client drive crawls live, query every UI surface, run single or bulk exports, save / open encrypted project snapshots, fetch GSC/GA4 data on demand, and modify settings — every action a human user can take in the desktop is callable from an agent. Crawl-completion webhooks, OS notifications, 22 built-in reports (histograms, top/bottom URLs, link positions, top words, cross-source orphans), per-URL Duplicates view, and a custom-CSS theme override round out the suite. Everything runs fully local — no telemetry, no cloud, MIT-style license.

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

chmod +x FreeCrawl-SEO-Tool-Start.sh
./FreeCrawl-SEO-Tool-Start.sh

# 1. Install prerequisites (skip any you already have)
brew install node@22 git
xcode-select --install      # Command Line Tools — required once

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

xattr -cr "/Applications/FreeCrawl SEO.app"

# 1. Install Node.js 22 LTS (Debian / Ubuntu via NodeSource)
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs git

# Fedora / RHEL : sudo dnf install nodejs:22 git
# Arch          : sudo pacman -S nodejs npm git

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

npm run build:cli
node apps/cli/dist/index.js https://example.com --depth 2 --max 500 --out out.csv
node apps/cli/dist/index.js --list urls.txt --out out.json     # list mode + JSON

# Log file analysis — parse an access log into the project for crawl × log reporting
node apps/cli/dist/index.js analyze-logs samples/apache-access.txt --project crawl.seoproject
node apps/cli/dist/index.js analyze-logs access.log --format iis-w3c --verify-bots --json

npm run build                                  # all packages + desktop + CLI
npm --workspace apps/desktop run build:win     # Windows installer (NSIS) + portable .exe
npm --workspace apps/desktop run build:mac     # macOS DMG (arm64 + x64)
npm --workspace apps/desktop run build:linux   # AppImage / .deb / .rpm

npm run build:mcp

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js"]
    }
  }
}

claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js

node apps/mcp-server/dist/index.js

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/path/to/apps/mcp-server/dist/index.js"],
      "env": { "FREECRAWL_PROJECT": "/path/to/audit.seoproject" }
    }
  }
}

node --version    # v22.x.x or v24.x.x
npm --version     # 10+

FreeCrawl-SEO-Tool/
├── 📄 FreeCrawl-SEO-Tool-Start.bat   # Windows one-click launcher
├── 📄 FreeCrawl-SEO-Tool-Start.sh    # macOS / Linux one-click launcher
├── 📄 CHANGELOG.md                   # versioned release notes
├── 📂 apps/
│   ├── 🪟 desktop/                   # Electron app (main + preload + renderer)
│   ├── ⌨  cli/                       # headless Node CLI
│   └── 🤖 mcp-server/                # MCP server for AI agents
└── 📂 packages/
    ├── 🔗 shared-types/              # IPC + domain types
    ├── 💾 db/                        # ProjectDb (node:sqlite) + migrations
    └── 🕷 core/                      # crawler engine (UI-agnostic)

Dependency graph

graph LR
  A[shared-types] --> B[db]
  B --> C[core]
  C --> D[desktop]
  C --> E[cli]
  B --> F[mcp-server]