🤖 Reddit Scraper

Modular Reddit data collection framework
Scrape subreddits, posts, and users into clean structured JSON.

A modular Reddit scraping pipeline designed for data collection, analytics, and research workflows.

The project gathers structured data about:

• 📚 Subreddits
• 📝 Posts
• 👤 Users

and exports everything as clean JSON datasets ready for:

• databases
• machine learning pipelines
• analytics
• data exploration

No manual scraping steps required.

• Modular scraper architecture
• Structured JSON output
• Automated scraping workflow
• MongoDB import helpers
• Large dataset handling utilities
• Environment-based configuration

run.py
│
├── subreddits.py
├── posts.py
└── users.py
↓
JSON datasets
↓
(optional) MongoDB import

Each scraper is independent and reusable.

git clone https://github.com/glowfi/reddit-scraper
cd reddit-scraper

python -m venv env
source env/bin/activate      # Linux / macOS
# env\Scripts\activate       # Windows

pip install -r requirements.txt

Edit env-sample and rename it:

.env

username=<RedditUsername>
password=<RedditPassword>
client_id=<ClientID>
client_secret=<ClientSecret>

TOTAL_SUBREDDITS_PER_TOPICS=6
SUBREDDIT_SORT_FILTER="hot"
POSTS_PER_SUBREDDIT=10
POSTS_SORT_FILTER="new"

Create Reddit API credentials here:

👉 https://www.reddit.com/prefs/apps

python run.py

Pipeline execution:

1. Scrape subreddits
2. Scrape posts
3. Scrape users
4. Export JSON datasets
5. Optional dataset splitting

JSON files are large (16–25MB). Download instead of viewing in browser.

Sample: https://files.catbox.moe/r7a7um.json

Sample: https://files.catbox.moe/5cf2xw.json

Sample: https://files.catbox.moe/yp506n.json

reddit-scraper/
├── subreddits.py
├── posts.py
├── users.py
├── run.py
├── utils/
│   ├── split.py
│   └── import_data_to_mongodb.sh
└── output/

After scraping:

./utils/import_data_to_mongodb.sh

Ensure MongoDB is running beforehand.

• Reddit API rate limits apply
• Scraping speed depends on network/API limits
• Designed for research & data workflows
• Respect Reddit API terms of service

Contributions, improvements, and issue reports are welcome.

Small focused PRs are preferred.

GPL-3.0