llama.cpp + TurboQuant+

A fork of ggml-org/llama.cpp integrating the TurboQuant+ implementation of TheTom's TurboQuant+ codec. This fork adds KV-cache and weight quantization types, cross-backend kernel support, and model-family-specific fixes.

Key insight: Asymmetric K/V compression — V tolerates aggressive compression, K does not. Always keep K at higher precision (f16 or q8_0) and compress V.

Recommended configs:

• Safe start: --cache-type-k f16 --cache-type-v turbo4
• Default: --cache-type-k q8_0 --cache-type-v turbo3
• Long context: --cache-type-k q8_0 --cache-type-v turbo2

llama-quantize model.f16.gguf model.tq4_1s.gguf TQ4_1S

• Auto-asymmetric K/V — complementary codec selection when both sides are turbo/TQ types
• Boundary V (layer-aware) — auto-enabled for turbo2-V; protects sensitive layers
• Sparse V dequantization — Metal-only; skips dequant for low-attention positions
• Flash Attention — auto-enabled with backend-specific kernels

• Gemma 4 — MoE token routing, op-concurrency handling
• Large MoE — up to 256-expert routing kernels
• Hybrid architectures (Mamba/GDN) — speculative decoding integration
• All existing llama.cpp models remain fully supported

# Clone this fork (not upstream!)
git clone https://github.com/eyalezer/llama.cpp.git

cd llama.cpp

# Build with your preferred backend
cmake -B build -DGGML_METAL=ON && cmake --build build -j
# or: cmake -B build -DGGML_CUDA=ON && cmake --build build -j
# or: cmake -B build -DGGML_VULKAN=ON && cmake --build build -j

# Run a model with turbo KV cache
llama-cli -m model.gguf --cache-type-k q8_0 --cache-type-v turbo3 -p "Hello!"

TurboQuant+ extends Google's original TurboQuant (ICLR 2026) paper with:

• Asymmetric K/V policy (paper)
• Layer-aware Boundary V protection (paper)
• Attention-gated sparse V dequantization (paper)
• TQ3_1S / TQ4_1S weight quantization (paper)
• Cross-backend kernel coverage (CUDA dp4a, HIP RDNA/CDNA, Vulkan coopmat, Metal V2.1)

The trailing + denotes ongoing extension work; the original TurboQuant codec remains the foundation.

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

VLM session with llama cli

Built-in web UI against llama serve

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

• cli
• completion
• server
• GBNF grammars

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain