A comprehensive application for downloading, processing, and storing cryptocurrency trade data from Bybit. The application provides both a command-line interface and a web-based user interface for managing and analyzing cryptocurrency data.

• Data Downloader: Automatically downloads cryptocurrency trade data from Bybit
• Data Processor: Processes raw trade data into aggregated minute-level data
• Data Storage: Stores processed data in a database for efficient retrieval
• Web Interface: Provides a user-friendly web interface for data visualization and management
• RESTful API: Offers a comprehensive API for accessing data and managing the application
• Scheduled Operations: Supports scheduled data downloads and processing
• Data Cleanup: Automatically cleans up old data based on retention policies

The application consists of the following main components:

1. Downloader (app/downloader/): Downloads cryptocurrency trade data from Bybit. Now modularized with BybitClient, ProgressManager, and FileValidator.
2. Processor (app/processor/): Processes raw trade data into aggregated minute-level data. Now modularized with CsvReader, TradeParser, NumbaAggregator, and ResultSaver.
3. Storage (app/storage/): Manages data storage and retrieval.
4. Web Interface (app/web/): Provides a web-based user interface. API endpoints are now split into job_api, download_api, data_api, and settings_api.
5. Application Manager (app/manager.py): Coordinates all components and manages scheduled operations.
6. CLI (app/cli/): Command-line interface components, including ArgParser and CliRunner.

• Python 3.9 or higher
• PostgreSQL (optional, for production deployment)
• Docker and Docker Compose (optional, for containerized deployment)

1. Clone the repository:

   git clone https://github.com/yourusername/cryptocurrency-data-processor.git
   cd cryptocurrency-data-processor

2. Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Initialize the database:

   python -m app.main --web-only

1. Clone the repository:

   git clone https://github.com/yourusername/cryptocurrency-data-processor.git
   cd cryptocurrency-data-processor

2. Build and run the application:

   docker-compose up -d

This will start the web interface on port 5000 and initialize the database.

The application provides a comprehensive command-line interface for managing data processing operations.

python -m app.main --full-pipeline

This will download, process, and store data for all cryptocurrencies.

# Download data only
python -m app.main --download-only

# Process data only
python -m app.main --process-only

# Run storage operations only (triggers data cleanup)
python -m app.main --storage-only

# Run web interface only
python -m app.main --web-only

# Download data for a specific cryptocurrency
python -m app.main --download-only --crypto BTCUSD

# Process data for a specific cryptocurrency
python -m app.main --process-only --crypto BTCUSD

# Run full pipeline for a specific cryptocurrency
python -m app.main --full-pipeline --crypto BTCUSD

# Use custom data directory
python -m app.main --data-dir /path/to/data

# Use custom number of download threads
python -m app.main --threads 20

# Use custom base URL
python -m app.main --base-url https://custom-url.com

# Enable verbose logging
python -m app.main --verbose

The web interface provides a user-friendly way to manage and visualize cryptocurrency data.

1. Start the web interface:

   python -m app.main --web-only

2. Open your browser and navigate to http://localhost:5000

The web interface includes:

• Dashboard: Overview of data statistics and application status
• Data Visualization: Charts and graphs for analyzing cryptocurrency data
• Data Management: Tools for downloading, processing, and cleaning up data
• Settings: Configuration options for the application

The application provides a comprehensive RESTful API for accessing data and managing the application. See API.md for detailed documentation.

# Get available cryptocurrencies
curl http://localhost:5000/api/coins

# Get raw trade data
curl "http://localhost:5000/api/trades?symbol=BTCUSD&start_time=2023-01-01T00:00:00&end_time=2023-01-02T00:00:00"

# Get aggregated data
curl "http://localhost:5000/api/aggregates?symbol=BTCUSD&start_time=2023-01-01T00:00:00&end_time=2023-01-02T00:00:00"

# Trigger data download
curl -X POST -H "Content-Type: application/json" -d '{"symbol": "BTCUSD"}' http://localhost:5000/api/download

# Get application statistics
curl http://localhost:5000/api/statistics

# Get application settings
curl http://localhost:5000/api/settings

# Update application settings
curl -X POST -H "Content-Type: application/json" -d '{"download_threads": 20, "data_retention_days": 60}' http://localhost:5000/api/settings

# Start a job
curl -X POST http://localhost:5000/api/job/download/start

# Get job statuses
curl http://localhost:5000/api/jobs

The application can be configured through environment variables or the config/config.py file.

• DATA_DIR: Directory to save downloaded data (default: "data")
• DATABASE_URL: Database connection URL (default: "sqlite:///cryptocurrency_data.db")
• BASE_URL: Base URL for downloading data (default: "https://public.bybit.com/trading")
• DEFAULT_DOWNLOAD_THREADS: Number of download threads (default: 10)
• LOG_LEVEL: Logging level (default: "INFO")
• DEFAULT_DATA_RETENTION_DAYS: Number of days to keep data (default: 30)

The config/config.py file contains all configuration settings. You can modify this file to customize the application behavior.

The application supports scheduled data downloads and processing. By default, data is downloaded at 2:00 AM and processed at 3:00 AM daily.

To enable scheduled operations in Docker:

docker-compose --profile scheduler up -d

To customize the schedule:

1. Modify the scheduler_interval, download_time, and process_time settings in config/config.py
2. Restart the application

Run the test suite:

python -m pytest tests/

Run integration tests:

python -m pytest tests/test_integration.py

The application downloads raw trade data with the following structure:

• timestamp: Trade timestamp
• symbol: Cryptocurrency symbol (e.g., 'BTCUSD')
• side: Trade side ('Buy' or 'Sell')
• size: Trade size
• price: Trade price
• tick_direction: Tick direction indicator

The application processes raw trade data into aggregated minute-level data with the following structure:

• timestamp: Minute timestamp
• symbol: Cryptocurrency symbol
• buy_volume: Total buy volume for the minute
• sell_volume: Total sell volume for the minute
• net_volume: Net volume (buy_volume - sell_volume) for the minute
• open_price: Opening price (first trade price) for the minute
• high_price: Highest price in the minute
• low_price: Lowest price in the minute
• close_price: Closing price (last trade price) for the minute

1. Database Connection Errors

   • Ensure the database is running and accessible
   • Check the database connection string in config/config.py

2. Download Failures

   • Check your internet connection
   • Verify the base URL in config/config.py
   • Ensure the cryptocurrency symbol is valid

3. Processing Errors

   • Ensure there is sufficient disk space
   • Check that the downloaded files are not corrupted
   • Verify the data directory permissions

4. Web Interface Issues

   • Ensure the port is not already in use
   • Check the firewall settings
   • Verify the Flask application is running

The application logs are stored in the logs directory. You can adjust the log level in config/config.py.

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Run the test suite
6. Submit a pull request

This project is licensed under the MIT License. See the LICENSE file for details.

For support, please open an issue on the GitHub repository or contact the development team.