Have a natural, spoken conversation with an AI!

This project lets you chat with a Large Language Model (LLM) using just your voice, receiving spoken responses in near real-time. Think of it as your own digital conversation partner.

(early preview - first reasonably stable version)

❗ Project Status: Community-Driven

This project is no longer being actively maintained by me due to time constraints. I've taken on too many projects and I have to step back. I will no longer be implementing new features or providing user support.

I will continue to review and merge high-quality, well-written Pull Requests from the community from time to time. Your contributions are welcome and appreciated!

A sophisticated client-server system built for low-latency interaction:

1. 🎙️ Capture: Your voice is captured by your browser.
2. ➡️ Stream: Audio chunks are whisked away via WebSockets to a Python backend.
3. ✍️ Transcribe: RealtimeSTT rapidly converts your speech to text.
4. 🤔 Think: The text is sent to an LLM (like Ollama or OpenAI) for processing.
5. 🗣️ Synthesize: The AI's text response is turned back into speech using RealtimeTTS.
6. ⬅️ Return: The generated audio is streamed back to your browser for playback.
7. 🔄 Interrupt: Jump in anytime! The system handles interruptions gracefully.

• Fluid Conversation: Speak and listen, just like a real chat.
• Real-Time Feedback: See partial transcriptions and AI responses as they happen.
• Low Latency Focus: Optimized architecture using audio chunk streaming.
• Smart Turn-Taking: Dynamic silence detection (turndetect.py) adapts to the conversation pace.
• Flexible AI Brains: Pluggable LLM backends (Ollama default, OpenAI support via llm_module.py).
• Customizable Voices: Choose from different Text-to-Speech engines (Kokoro, Coqui, Orpheus via audio_module.py).
• Web Interface: Clean and simple UI using Vanilla JS and the Web Audio API.
• Dockerized Deployment: Recommended setup using Docker Compose for easier dependency management.

• Backend: Python < 3.13, FastAPI
• Frontend: HTML, CSS, JavaScript (Vanilla JS, Web Audio API, AudioWorklets)
• Communication: WebSockets
• Containerization: Docker, Docker Compose
• Core AI/ML Libraries:
  • RealtimeSTT (Speech-to-Text)
  • RealtimeTTS (Text-to-Speech)
  • transformers (Turn detection, Tokenization)
  • torch / torchaudio (ML Framework)
  • ollama / openai (LLM Clients)
• Audio Processing: numpy, scipy

This project leverages powerful AI models, which have some requirements:

• Operating System:
  • Docker: Linux is recommended for the best GPU integration with Docker.
  • Windows: The provided script (install.bat) provides automated setup.
  • macOS: Native support with optimized installation scripts and configuration.
  • Manual: Manual steps are possible on all platforms but may require more troubleshooting (especially for DeepSpeed).
• 🐍 Python: 3.9 or higher (if setting up manually).
• 🚀 GPU: A powerful CUDA-enabled NVIDIA GPU is highly recommended, especially for faster STT (Whisper) and TTS (Coqui). Performance on CPU-only or weaker GPUs will be significantly slower.
  • The setup assumes CUDA 12.1. Adjust PyTorch installation if you have a different CUDA version.
  • Docker (Linux): Requires NVIDIA Container Toolkit.
• 🐳 Docker (Optional but Recommended): Docker Engine and Docker Compose v2+ for the containerized setup.
• 🧠 Ollama (Optional): If using the Ollama backend without Docker, install it separately and pull your desired models. The Docker setup includes an Ollama service.
• 🔑 OpenAI API Key (Optional): If using the OpenAI backend, set the OPENAI_API_KEY environment variable (e.g., in a .env file or passed to Docker).

Clone the repository first:

git clone https://github.com/KoljaB/RealtimeVoiceChat.git
cd RealtimeVoiceChat

Now, choose your adventure:

curl -sSL https://raw.githubusercontent.com/andy-aimer/RealtimeVoiceChat/main/deployment/macos/quick_install_macos.sh | bash

alias rtvc="cd ~/RealtimeVoiceChat && source venv_macos/bin/activate"
alias start-rtvc="cd ~/RealtimeVoiceChat && ./start_macos.sh"
alias start-rtvc-ssl="cd ~/RealtimeVoiceChat && ./start_macos_ssl.sh"

If using Docker: Your application is already running via docker compose up -d! Check logs using docker compose logs -f app.

If using Manual/Script Installation:

1. Activate your virtual environment (if not already active):

   # Linux/macOS: source ../venv/bin/activate
   # Windows: ..\venv\Scripts\activate

2. Navigate to the code directory (if not already there):

   cd code

3. Start the FastAPI server:

   python server.py

Accessing the Client (Both Methods):

1. Open your web browser to http://localhost:8000 (or your server's IP if running remotely/in Docker on another machine).
2. Grant microphone permissions when prompted.
3. Click "Start" to begin chatting! Use "Stop" to end and "Reset" to clear the conversation.

Want to tweak the AI's voice, brain, or how it listens? Modify the Python files in the code/ directory.

⚠️ Important Docker Note: If using Docker, make any configuration changes before running docker compose build to ensure they are included in the image.

• TTS Engine & Voice (server.py, audio_module.py):

  • Change START_ENGINE in server.py to "coqui", "kokoro", or "orpheus".
  • Adjust engine-specific settings (e.g., voice model path for Coqui, speaker ID for Orpheus, speed) within AudioProcessor.__init__ in audio_module.py.

• LLM Backend & Model (server.py, llm_module.py):

  • Set LLM_START_PROVIDER ("ollama" or "openai") and LLM_START_MODEL (e.g., "hf.co/..." for Ollama, model name for OpenAI) in server.py. Remember to pull the Ollama model if using Docker (see Installation Step A3).
  • Customize the AI's personality by editing system_prompt.txt.

• STT Settings (transcribe.py):

  • Modify DEFAULT_RECORDER_CONFIG to change the Whisper model (model), language (language), silence thresholds (silence_limit_seconds), etc. The default base.en model is pre-downloaded during the Docker build.

• Turn Detection Sensitivity (turndetect.py):

  • Adjust pause duration constants within the TurnDetector.update_settings method.

• SSL/HTTPS (server.py):

  • Set USE_SSL = True and provide paths to your certificate (SSL_CERT_PATH) and key (SSL_KEY_PATH) files.
  • Docker Users: You'll need to adjust docker-compose.yml to map the SSL port (e.g., 443) and potentially mount your certificate files as volumes.
  Generating Local SSL Certificates (Windows Example w/ mkcert)

  1. Install Chocolatey package manager if you haven't already.
  2. Install mkcert: choco install mkcert
  3. Run Command Prompt as Administrator.
  4. Install a local Certificate Authority: mkcert -install
  5. Generate certs (replace your.local.ip): mkcert localhost 127.0.0.1 ::1 your.local.ip * This creates .pem files (e.g., localhost+3.pem and localhost+3-key.pem) in the current directory. Update SSL_CERT_PATH and SSL_KEY_PATH in server.py accordingly. Remember to potentially mount these into your Docker container.

Automatic hardware protection for long-running deployments on Raspberry Pi 5.

The system now includes built-in thermal monitoring to protect your hardware from overheating during extended use. When CPU temperature exceeds safe thresholds, the system automatically reduces workload to prevent thermal throttling and potential hardware damage.

How It Works:

• Monitors CPU temperature continuously (every 5 seconds by default)
• Triggers protection at 85°C - pauses LLM inference to reduce CPU load
• Resumes normal operation at 80°C (5°C hysteresis prevents oscillation)
• Displays visual warning in UI when protection is active
• Logs structured events for monitoring and debugging

Environment Variables:

Configure thermal thresholds via environment variables (add to .env file or Docker environment):

# Thermal Protection Configuration
THERMAL_TRIGGER_THRESHOLD=85.0    # Temperature (°C) to trigger protection (default: 85.0)
THERMAL_RESUME_THRESHOLD=80.0     # Temperature (°C) to resume normal operation (default: 80.0)
THERMAL_CHECK_INTERVAL=5.0        # Seconds between temperature checks (default: 5.0)

Platform Support:

• Raspberry Pi 5: Full thermal monitoring via /sys/class/thermal/thermal_zone0/temp
• Other platforms (macOS/Windows/Linux): Gracefully disabled (returns -1, no errors)

Docker Configuration:

Add thermal environment variables to docker-compose.yml:

services:
  app:
    environment:
      - THERMAL_TRIGGER_THRESHOLD=85.0
      - THERMAL_RESUME_THRESHOLD=80.0
      - THERMAL_CHECK_INTERVAL=5.0

Monitoring Thermal State:

Check thermal status via the health endpoint:

curl http://localhost:8000/health

Response includes thermal state:

{
  "status": "healthy",
  "thermal": {
    "temperature": 75.2,
    "protection_active": false,
    "inference_paused": false,
    "trigger_threshold": 85.0,
    "resume_threshold": 80.0
  }
}

Testing on Pi 5:

Simulate thermal load using stress-ng:

# Install stress-ng
sudo apt-get install stress-ng

# Generate CPU load to trigger thermal protection
stress-ng --cpu 4 --timeout 60s --metrics-brief

Monitor thermal state in logs or UI banner during stress testing.

Got ideas or found a bug? Contributions are welcome! Feel free to open issues or submit pull requests.

The core codebase of this project is released under the MIT License (see the LICENSE file for details).

This project relies on external specific TTS engines (like Coqui XTTSv2) and LLM providers which have their own licensing terms. Please ensure you comply with the licenses of all components you use.