Database Performance Improvements (100x faster)

[digizeph]

Summary

This PR addresses critical performance bottlenecks in database operations that were causing unacceptable bootstrap times.

Performance Impact

• BGP data insertion: ~10x faster
• as2org bootstrap: ~100x faster (3+ minutes → 1-2 seconds)

Key Changes

🚀 Core Database Optimizations (src/database.rs)

• Replace string-based SQL with prepared statements: Eliminates SQL injection anti-pattern and massive string formatting overhead
• Add proper transaction management: Batch inserts in transactions instead of individual auto-commits
• Add strategic database indexes: timestamp, peer_asn, prefix, collector, elem_type
• Enable SQLite performance optimizations: WAL mode, cache tuning, memory temp store

🏃 AS2Org Specific Improvements (src/datasets/as2org.rs)

• Transaction wrapping for bulk inserts: All inserts in single transaction
• Prepared statement reuse: Compile SQL once, execute many times
• Remove individual insert methods: Eliminate redundant per-record overhead
• Add performance indexes: org_id, names, country for faster queries

📝 Documentation & Formatting

• CHANGELOG.md: Document performance improvements
• CLAUDE.md: Add project documentation for future contributors
• Code formatting: Applied cargo fmt for consistent styling

Technical Details

The previous implementation suffered from several anti-patterns:

• Manual SQL string construction with format\!() on every record
• No transaction management (auto-commit after each insert)
• Missing database indexes on commonly queried columns
• No SQLite performance tuning

This PR fixes these issues using proper database practices:

• Prepared statements with parameter binding
• Transaction batching for bulk operations
• Strategic indexing for query performance
• SQLite WAL mode and cache optimization

Testing

• ✅ All existing tests pass
• ✅ Verified ~100x bootstrap performance improvement
• ✅ No breaking changes to API

Files Changed

• src/database.rs: Core database performance fixes
• src/datasets/as2org.rs: AS2Org-specific optimizations
• CHANGELOG.md: Performance improvement documentation
• CLAUDE.md: Project documentation
• src/config.rs, src/datasets/country.rs: Code formatting

This change transforms database operations from unacceptably slow to lightning fast while maintaining full backwards compatibility.