If I understood well the main scope of the project is to allow standard PC with low VRAM to run big models with MoE architecture (the current standard). The idea behind this is to perform all the main computation using only GPU and VRAM. But not using quantized weights and cache the possibility to run effectively big models is still limited. To improve significantly the inference speed it is necessary to use both quantized weights and cache. Suppose for example to run Qwen3.8-Flash-Next, this is a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings (that can be stored in the RAM), with 6B parameters activated per token (only these must be located in VRAM). With q4_0 quantization the VRAM usage of the selected experts will require about 3GB of VRAM. Using a PC with 12GB VRAM, this imply that will be enough space to store about 8GB of cache (good but not enough for coding agent), while using q4_0 quantized cache it will be possible to store in VRAM the equivalent of about 32GB of uncompressed cache. This is the difference between having a toy model and a working model.
If I understood well the main scope of the project is to allow standard PC with low VRAM to run big models with MoE architecture (the current standard). The idea behind this is to perform all the main computation using only GPU and VRAM. But not using quantized weights and cache the possibility to run effectively big models is still limited. To improve significantly the inference speed it is necessary to use both quantized weights and cache. Suppose for example to run Qwen3.8-Flash-Next, this is a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings (that can be stored in the RAM), with 6B parameters activated per token (only these must be located in VRAM). With q4_0 quantization the VRAM usage of the selected experts will require about 3GB of VRAM. Using a PC with 12GB VRAM, this imply that will be enough space to store about 8GB of cache (good but not enough for coding agent), while using q4_0 quantized cache it will be possible to store in VRAM the equivalent of about 32GB of uncompressed cache. This is the difference between having a toy model and a working model.