BenchPup is a local, desktop-first benchmark recorder for evaluating LLM coding and code-review tasks. Detailed benchmark runs, historical scoreboard entries, profiles, prompts, and reviews are stored in SQLite.
Version 0.4.1 Alpha.
Completed:
- interactive screen-based CLI
- CSV and hardware-profile imports
- scoreboard browsing, legacy HTML viewing, and standalone HTML analytics
- JSON backup and restore
- curated JSONL Dataset Builder with filters, redaction, manifests, and validation
- descriptive statistics, model/session comparisons, and UTC trend reports
Phase 4 reporting and analytics are complete. Next: the PySide6 GUI, which will reuse the existing engine/reporting boundaries.
python src/main.pyChoose an action from the interactive menu. Use the Dataset Builder to preview, build, and validate curated JSONL datasets from detailed BenchmarkRun records.
Python 3.11+
pyreadline3>=3.5.4; sys_platform == "win32"
prompt_toolkit>=3.0.0
python src/main.py
python -m unittest discover -s tests -v