model: add Kimi-K3 text model

[pwilkin]

Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five things that architecture does not have:

1. cross-layer residual attention (attn_res_block_size)
2. latent MoE (routed experts run at n_expert_latent)
3. situ activation (replaces SwiGLU everywhere)
4. MLA output gate (sigmoid gate before o_proj)
5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b)

Reuses DeepSeek4's HC_PRE for the cross-layer residual weighted sum. Supports repack for the MXFP4 weights in conversion.

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, ran the conversion session with Opus.

Now need someone to actually convert and test :)

[ngxson]

may need to shorten comments too, IMO some/most comments are too verbose

[ngxson]

I'd suggest renaming:

• ckpts --> resi (short for residual stream, same naming mentioned in the paper)
• drop the _cache since technically there is no cache here, resi_stack should be enough

also, it might be cleaner if these are grouped into a new struct and explicitly pass it like this:

struct attn_resi; // private struct, defined inside cpp file
void res_push(attn_resi r, int64_t n_embd, int64_t n_tokens);
ggml_tensor * res_stack(attn_resi r, int64_t n_embd, int64_t n_tokens);

[ngxson]

these might not be necessary, I suppose for compat?

[pwilkin]

Yeah, those are artifacts from the mock model runs, for the real model I'll purge them.

[Green-Sky]

https://huggingface.co/inference-optimization/Kimi-K3-0.18B

potentially useful, depending on how faithful the model is reconstructed in 0.18B .

[pwilkin]

@Green-Sky I wouldn't put it up without checking parity with a mock model 😄

[Green-Sky]

@Green-Sky I wouldn't put it up without checking parity with a mock model 😄

Isnt that a mock model?

[pwilkin]

Yeah, that's what I'm saying, already built one of my own for parity testing purposes when doing the PR.

[ngxson]

@GrEarl please put your comment inside a collapsible block, it takes up too much space & make the discussion hard to keep track

[GrEarl]

Sorry for the long text. reposting it collapsed and trimmed.

Ran this branch against the actual Kimi-K3 checkpoint. One thing needs fixing; the rest is context.

LLAMA_MAX_EXPERTS is 512, Kimi-K3 has 896

src/llama-hparams.h:11    #define LLAMA_MAX_EXPERTS 512 // Qwen3 Next
src/llama-model.cpp:1117  GGML_ASSERT(hparams.n_expert <= LLAMA_MAX_EXPERTS);

Arch-generic path, right after LLM_KV_EXPERT_COUNT is read and before any arch hook. The converter itself prints gguf: expert count = 896, so a converted file aborts at load.

It cannot be worked around on the file side: understating expert_count contradicts ne[2] = 896 on the ffn_*_exps tensors.

LLAMA_MAX_EXPERTS sizes exactly one thing in the tree — cur_experts[LLAMA_MAX_EXPERTS] in build_moe_ffn (src/llama-graph.cpp:2128) — and only [0, n_expert_used) is ever touched, so 1024 costs 4 KB of stack.

--- a/src/llama-hparams.h
+++ b/src/llama-hparams.h
@@
 // bump if necessary
 #define LLAMA_MAX_LAYERS  512
-#define LLAMA_MAX_EXPERTS 512 // Qwen3 Next
+#define LLAMA_MAX_EXPERTS 1024 // Kimi-K3

Not verified: I have not confirmed a K3 GGUF actually loads with this — I do not have a machine that can hold 938 GiB. It is inferred from the assert and the single array it bounds. Whether to include it here or split it out is your call.

Dry run: prepare_tensors() completes, all 93 layers mapped, file type = 38

convert_hf_to_gguf.py --remote moonshotai/Kimi-K3 --dry-run --outtype auto, ~3 min on 8 vCPU / 32 GiB.

• hparams read correctly: context length 1048576 / embedding length 7168 / feed forward length 33792 / head count 96 / expert count 896 / experts used count 16 / expert score gating function sigmoid
• file type = 38 (MOSTLY_MXFP4_MOE), so the MXFP4 detection and the ftype override both take effect
• tensor map produced for all 93 layers with no unmapped names, including ssm_g / ssm_conv1d_{q,k,v} / ssm_{a,beta,dt,f_a,f_b,norm} / attn_res_score / ffn_res_score / output_res_score / ffn_routed_{down,up,norm} / exp_probs_b, and the MLA layers' attn_{q_a,q_b,kv_a_mqa,k_b,v_b,gate}
• the expert writing path returned without raising, so every (layer, w1|w2|w3) group had all 896 weight_packed/weight_scale pairs
• every *_res_norm / *_res_proj pair was fused, nothing left over

I did not write a complete GGUF or load one, so none of this says anything about inference correctness.

repack_mxfp4_blocks is bit-exact — 0 mismatches over 11,010,048 values

In --dry-run the expert tensors stay lazy, so the dry run never executes repack_mxfp4_blocks. I checked it separately.

Before seeing this PR I had derived the same repack from the compressed-tensors and ggml sources. Your output is byte-identical to mine, on layers.48.block_sparse_moe.experts.0.w1 (logical [3072, 3584]), fetching 5.6 MiB via HTTP Range:

mismatched elements : 0 / 11010048
max abs error       : 0.0
bytes in/out        : 5849088 / 5849088
bpw                 : 4.2500

Reference side: compressed-tensors nvfp4/helpers.py::pack_fp4_to_uint8 (element 2i in the low nibble, sign in bit 3, kE2M1ToFloat = [0, .5, 1, 1.5, 2, 3, 4, 6]) and mx_utils.py::decompress_mx_scale (2 ** (e - 127)). ggml side: dequantize_row_mxfp4 in ggml-quants.c, kvalues_fp4 in ggml-common.h, ggml_e8m0_to_fp32_half in ggml-impl.h (bits = (x - 1) << 23 for x >= 2).

So the docstring claim — kvalues doubled, scale halved, represented value unchanged — holds exactly. E8M0 exponents on this tensor are 120..122, so the x < 2 branch is not exercised by this data. Since the diff moves DeepSeek-V4's _pack_mxfp4_blocks into base.py and aliases it back, this covers that path too.

Reproduction, from a checkout of this branch (fetches ~5.6 MiB, stores no weights):

#!/usr/bin/env python3
import json
import urllib.request

import numpy as np
import torch

from conversion.base import repack_mxfp4_blocks

REPO = "moonshotai/Kimi-K3"
SHARD = "model-00049-of-000096.safetensors"
URL = f"https://huggingface.co/{REPO}/resolve/main/{SHARD}"
E2M1 = np.array([0.0, .5, 1., 1.5, 2., 3., 4., 6.], dtype=np.float32)
KV = np.array([0, 1, 2, 3, 4, 6, 8, 12, 0, -1, -2, -3, -4, -6, -8, -12], dtype=np.int8)


def rng(a, b):
    req = urllib.request.Request(URL, headers={"Range": f"bytes={a}-{b}"})
    return urllib.request.urlopen(req).read()


n = int.from_bytes(rng(0, 7), "little")
hdr = json.loads(rng(8, 8 + n - 1))
name = next(k for k in sorted(hdr) if k.endswith("experts.0.w1.weight_packed"))


def get(k):
    s, e = hdr[k]["data_offsets"]
    return np.frombuffer(rng(8 + n + s, 8 + n + e - 1), np.uint8).reshape(hdr[k]["shape"])


packed = get(name)
scale = get(name.replace("weight_packed", "weight_scale"))
rows, cols = packed.shape[0], packed.shape[1] * 2

# reference: compressed-tensors pack_fp4_to_uint8 / decompress_mx_scale, inverted
idx = np.empty((rows, cols), np.uint8)
idx[:, 0::2], idx[:, 1::2] = packed & 0xF, packed >> 4
mag = E2M1[(idx & 7).astype(np.int32)]
ref = np.where((idx & 8) != 0, -mag, mag).reshape(rows, cols // 32, 32) \
    * np.ldexp(np.ones(scale.shape, np.float32), scale.astype(np.int32) - 127)[:, :, None]

# ggml: dequantize_row_mxfp4 on the repacked blocks
blk = repack_mxfp4_blocks(torch.from_numpy(packed.copy()),
                          torch.from_numpy(scale.copy())).reshape(rows, -1, 17)
out = np.empty((rows, blk.shape[1], 32), np.float32)
out[:, :, :16] = KV[(blk[:, :, 1:] & 0xF)]
out[:, :, 16:] = KV[(blk[:, :, 1:] >> 4)]
got = out * np.ldexp(np.ones(blk.shape[:2], np.float32),
                     blk[:, :, 0].astype(np.int32) - 128)[:, :, None]

ref, got = ref.reshape(rows, cols), got.reshape(rows, cols)
print(f"{name}  [{rows}, {cols}]")
print(f"mismatched elements : {int((ref != got).sum())} / {ref.size}")
print(f"max abs error       : {float(np.abs(ref - got).max())}")
print(f"bytes in/out        : {packed.nbytes + scale.nbytes} / {blk.nbytes}")
print(f"bpw                 : {blk.nbytes * 8 / (rows * cols):.4f}")

--remote does not fetch the tokenizer files — not a problem with this PR

This is where my dry run stopped. Relevant only if you test with --remote.

Download patterns are ["LICENSE", "*.json", "*.md", "*.txt", "tokenizer.model"]. K3 keeps its vocabulary in tiktoken.model and its tokenizer class in tokenization_kimi.py (which relative-imports encoding_k3.py), so neither arrives, and AutoTokenizer.from_pretrained(self.dir_model, trust_remote_code=True) in kimi_linear.py fails with does not appear to have a file named tokenization_kimi.py. Kimi-K2 / K2.5 reach the same call through deepseek.py.

Measured with snapshot_download: adding *.model and *.py goes from 15 files / 57.05 MiB to 33 files / 59.89 MiB (+2.84 MiB) and brings in all three. Most of the 57 MiB is model.safetensors.index.json, already pulled by *.json. self.remote_hf_model_id is available on the model instance if letting transformers resolve the code from the Hub is preferable to widening the patterns.

With a complete local snapshot, set_vocab in this PR works as-is. What I checked:

• AutoTokenizer.from_pretrained(dir, trust_remote_code=True) returns TikTokenTokenizer
• model._mergeable_ranks has 163,584 entries and special_tokens 256; 163,584 + 256 = 163,840 = vocab_size exactly, so no UNUSED holes
• get_vocab_base_pre hashes to 81212dc7cdb7e0c1074ca62c5aeab0d43c9f52b8a737be7b12a777c953027890, which base.py maps to kimi-k2 — no new pre-tokenizer needed, as your comment says
• merge reconstruction yields 163,328 entries; tokens[163584] == "[BOS]", tokens[163839] == "[PAD]"
• tokenizer.eos_id is 163585 ([EOS]) while config.json and generation_config.json both say 163586 (<|end_of_msg|>), so the eos restore in set_vocab is doing real work

One note for anyone converting: tokenization_kimi.py imports tiktoken, and tiktoken.load.read_file requires blobfile even for a local path. Neither is in requirements/requirements-convert_hf_to_gguf.txt.

AI usage disclosure: I found this while building a quantized model with the help of AI. Because of that, some of my views here may be technically wrong, or wrong about how llama.cpp is implemented. I am also not fluent in English, so I used AI to help with the translation.

[thispwd]

Tested this branch today with a converted K3 checkpoint. Everything loaded successfully and generation worked as expected.

Conversion (conversion/kimi_k3.py)

• Dry run correctly detected the model (896 experts, MXFP4_MOE ftype).
• Full export from local safetensors completed cleanly, producing 2,573 tensors split across multiple output files.
• No manual fixes were required.

Runtime

• Running with hybrid GPU + CPU offload (-ngl 99 with -ot keeping part of the expert stacks in system RAM).
• Output is coherent and accurate on factual Q&A.
• Thinking segments are parsed correctly into reasoning_content via the OpenAI-compatible server.

Two observations that may help others:

1. -ot layer placement
   When using -ot to keep only a subset of MoE experts on the GPU, avoid contiguous layer ranges. Layers are distributed evenly across devices, so configurations like "layers 0–34 on GPU" can overload the first GPU(s). Interleaving layers (e.g. every Nth layer on the GPU) results in a much more balanced memory distribution.

2. XTML chat template
   The XTML chat template currently circulating uses namespace({...}), which Minja rejects with:

   namespace() arguments must be kwargs

   Changing it to namespace(key=value, ...) resolves the issue.

Thank you.

[usrlocalben]

@GrEarl 's late-edition Q2 works with the same template issue/workaround as @thispwd describes.
edit: I now see that the Q2 had its 1-of-N GGUF shard updated with an improved chat template.

There are stray template elements (or special tokens?) in the output, but the reasoning parser still detects begin/end and eos token ends the turn properly.

e.g.

<|open|>think<|sep|>The user is asking me to "tell the one about
the cat and the fire extinguisher again" — as if we've had a prior
conversation where I told a story or joke about a cat and a fire extinguisher.

---snip---

If you remember details from the original, tell me and I'll work
them in — or I can spin a completely different take.<|close|>response<|sep|><|close|>message<|sep|>

(newlines added to avoid side-scrolling)

It would be helpful if llama warmed up the experts. It doesn't anymore, or doesn't with this model. The warmup run does not force all experts on. I know this is also a problem for DSv4 warmup.

Many long stories about cats or doubly-linked list impls are needed to get 2^7*7 mmap experts warmed up.

The no-imatrix (I think I can safely assume) Q2 quant is coherent, and seems quite capable at a glance.

[csabakecskemeti]

FYI: Here's another chat template PR on HF: https://huggingface.co/moonshotai/Kimi-K3/discussions/66

[usrlocalben]

FYI: Here's another chat template PR on HF: https://huggingface.co/moonshotai/Kimi-K3/discussions/66

This is the template I used.

[csabakecskemeti]

I can also confirm the conversion and the chat template gives coherent answer
Full precision cmoe mmap (system: 128gb vram, 1tb ram, experts loaded from nvme)

[SolshineCode]

Tested kimi-k3-text @ cf11c4c end to end with a synthetic fixture. Greedy outputs from llama-server match the HF reference implementation exactly, token for token, on all 3 test prompts.

Method: I generated a random-init shrunk K3 (90M params, f32) that keeps the full structure: hybrid KDA/MLA with the real linear_attn_config schedule shape (full attn every 4th layer plus the last), latent MoE with 16 experts and routed_expert_hidden_size, attention-residual blocks, situ, the MLA output gate, the full-rank KDA gate, and the real tokenizer. One honest caveat about what "reference" means here: the released modeling code can't run on CPU as shipped (fla's KDA kernels are Triton-only), so I rerouted them to fla's own naive torch ops (naive_recurrent_kda and friends). The match is against fla's reference math, not the fused kernels. Comparison was done by POSTing raw input_ids to /completion with top_k=1, which validates the graph independent of tokenization, and the decoded text matched too.

Hardware is deliberately ancient: 2x Xeon E5-2609v2 (AVX only, no AVX2), 512GB DDR3, CPU-only build so far. Build is clean on this ISA, conversion works (tensor mapping, res_norm x res_proj fusion, expert merge, tokenizer; transformers git-main / 5.15.0.dev0), and llama-server generates at ~55 tok/s on the tiny model.

Two problems hit along the way:

1. llama-cli hangs silently in --no-conversation mode (repro 3/3, with and without stdin closed; conversation mode is fine). Might not be K3-specific.
2. convert_hf_to_gguf breaks on transformers >= 5.15: bytes_to_unicode moved out of models.gpt2.tokenization_gpt2 (it's in convert_slow_tokenizer now). A one-line try/except in conversion/qwen.py fixes it. Happy to PR that separately.

Fixture generator, CPU shim, and reference outputs: https://gist.github.com/SolshineCode/3115760b0c3b655563a3102ba897c426. I can upstream the fixture into the test suite if useful, or defer to @200lz if their fixtures already cover this. Once a real quant exists I can also run a full-scale streamed validation on this box (512GB RAM, mmap + -ot exps=CPU), and it has sm_52 Teslas if legacy-CUDA coverage is ever wanted.

AI usage disclosure: the test harness and this report were built with Claude running on my machine; the numbers are from real runs I can rerun on request.

[200lz]

@SolshineCode Excellent validation—thank you for sharing the fixture and the CPU reference path.

My work does not duplicate your end-to-end fixture. I analyzed the released checkpoint schema using all 96 official safetensors headers, without downloading tensor payloads. The production checkpoint confirms:

• layer 0 is dense, while layers 1–92 are routed MoE;
• every MoE layer contains the exact expert set 0..895;
• KDA layers are 0–2, 4–6, ... , 88–90;
• MLA layers are 3, 7, ... , 87, 91, 92;
• the routed-expert inventory contains exactly 92 × 896 × 3 = 247,296 MXFP4 block/scale pairs;
• Attention Residual uses four per-layer tensors, two output tensors, and 12-layer block boundaries.

Please do not defer the execution fixture to me—your fixture covers an area I have not implemented and would be valuable upstream.

I can instead review the PR against the full released tensor vocabulary and contribute a compact schema-level regression test, if useful, covering the exceptional final MLA layer, the dense-to-MoE boundary, expert-set invariants, and self_attn.g_proj mapping.

I’ll first inspect the current PR branch to avoid duplicating existing tests.

[SolshineCode]

Ran the fixture against the CUDA build as well: kimi-k3-text @ cf11c4c, CUDA 12.4 with -DCMAKE_CUDA_ARCHITECTURES=52, 2x Tesla M40 (compute 5.2), -ngl 99 --tensor-split 1,1. All 3 prompts match the reference token for token, ~109 t/s vs ~55 CPU-only. One flag: load prints resolve_fused_ops: layer 3 is assigned to device CUDA0 but Flash Attention is assigned to device CPU (usually due to missing support), so at least one attention op lacks an sm_52 kernel. Logs: https://gist.github.com/SolshineCode/3115760b0c3b655563a3102ba897c426 (sm52_results.md)

@200lz thanks, schema-level checks are exactly what my fixture doesn't do, so those complement each other well. I cross-checked your header-derived layer map against the released config's linear_attn_config and they match exactly (your 0-indexed MLA 3, 7, ..., 91, 92 is the config's 1-based full_attn_layers). The fixture already encodes both structural exceptions you mention: it ends with consecutive MLA layers like 91+92, and it has the dense layer 0 to MoE boundary. I'll upstream it as an execution test to sit alongside your schema regression test, in whatever form the maintainers prefer.

[200lz]

@SolshineCode Thanks for cross-checking the released layer map and for confirming the two structural exceptions in the fixture.

That separation sounds ideal: your fixture can cover execution and token-level parity, while I’ll focus on compact schema regression coverage derived from the official checkpoint.

I’ll review the current PR tests and prepare the smallest non-duplicative schema test proposal, especially around the final MLA layer, the dense/MoE boundary, expert-set invariants, and tensor-name mapping.

[200lz]

@pwilkin I completed a read-only test-gap review of this PR against the full released K3 checkpoint schema.

The external execution fixture provides strong CPU/CUDA parity coverage, but the PR currently has no dedicated K3 schema regression test in CI.

I would like to contribute a compact converter-level test covering:

• the final consecutive MLA layers;
• layer 0 dense vs. layers 1+ MoE;
• official expert/shared-expert invariants;
• Attention Residual block grouping;
• and context-sensitive self_attn.g_proj mapping.

The test would use synthetic config/tensor metadata only—no model weights, runtime execution, or overlap with the existing fixture.

Would you prefer this added to #26185, or submitted as a small follow-up PR after merge?

[idumlupinar]

Can we run this text model on dual RTX 3090 and 128GB DDR4 memory and 5800x3d cpu on Windows 11?

[RodriMora]

Can we run this text model on dual RTX 3090 and 128GB DDR4 memory and 5800x3d cpu on Windows 11?

No, the lowest size, if by some miracle Aes or Bart make a Q1-2 with imatrix quant that is still coherent, that would even be 700GB

[csabakecskemeti]

Quality topic
Just an opinion based on my own quant, curious what's other's experience:
The intelligence per GB feels off with this model. I'm testing my Q2 quant with zero shot Mario clone, and it struggles to create a working (I've not meant errors in the html the game is not working) game. My reference is GLM-5.2 Q5 which has done it first shot at first try. I know the Q5 is less lossy than Q2 but at GB level it perform better. (At least in HTML coding)
This may an unfair comparison, or some issue with my quant. Otherwise the output is coherent and makes sense.

I also acknowledge that this is work in progress, also this maybe my local issue. Shared just for fyi

Looking forward to hear other's experience

[SolshineCode]

Okay so I Tried to reproduce the multi-turn corruption @Tonoken3 reported, using my tiny fixture so iteration is seconds instead of minutes (for testing). On cf67f0d I can't repro it at small scale: with confirmed partial prefix reuse (timings.prompt_n 53 vs 343 fresh), both single divergent-turn rollback and 6 turns of accumulated reuse give token-identical greedy outputs vs a fresh server, on CPU and on CUDA (compute 5.2). So at least on this branch tip the basic KDA rollback path looks right, and the remaining surface is long histories (state checkpoint exhaustion), specific serving flags, batch effects, newer CUDA archs, or something fixed since your test. Repro harness and results are in the gist: https://gist.github.com/SolshineCode/3115760b0c3b655563a3102ba897c426 (repro_kda_cache_reuse_chat.py, kda_cache_results.md), if useful for pinning it down on a rig that does repro instead of mine.
One separate small bug found while building this is that /completion returns HTTP 500 ("does not match the expected Content-only format") when the model emits invalid UTF-8 mid-stream or n_predict cuts a multibyte character, even with --no-jinja and --reasoning-format none. Random-weights models hit it constantly and low-bpw quants can emit malformed bytes too; a graceful fallback to raw content might beat a 500 on the raw endpoint.

[danielhanchen]

@pwilkin I added mmproj + some fixes to unslothai#48 which stacks on top of yours - feel free to merge commits / take them :)

[csabakecskemeti]

It has taken a while but the zeroshot 'supermario clone' successfully completed on the gguf converted full precision model. Immaculate execution!

[pwilkin]

@danielhanchen thanks :) the only question I have is about the conversion fixes - judging from this thread people have already converted the model and verified it works correctly using this code, so I'm wondering about those claims (the KV cache I'll look into).

[createthis]

I downloaded unsloth's UD-Q2_K_XL and started it up on commit cf67f0d of this branch.

My machine is 768gb RAM + 92gb VRAM (6000 pro).

Startup commands:

Details

echo 0 | sudo tee /proc/sys/kernel/numa_balancing
echo 3 | sudo tee /proc/sys/vm/drop_caches

./build/bin/llama-server \
    --model /data2/Kimi-K3-GGUF/UD-Q2_K_XL/Kimi-K3-UD-Q2_K_XL-00001-of-00019.gguf \
    --alias Kimi-K3:1.5t:UD-Q2_K_XL \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    -ot "blk\.(3|4)\.ffn_.*=CUDA0,exps=CPU" \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 1.0 \
    --top-p 1.0 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --port 11434

The first thing I noticed is that warmup seems broken. It doesn't fully populate system RAM until I prompt. I haven't tried master in a while so I don't know if that is specific to this branch/model, sorry.

It's pretty slow. I normally get about 20 tok/s eval with kimi 2.6 Q4. I'm seeing about 3 tok/s with K3 Q2:

1.36.989.062 I slot print_timing: id  3 | task 0 | prompt eval time =   21132.82 ms /    89 tokens (  237.45 ms per token,     4.21 tokens per second)
1.36.989.066 I slot print_timing: id  3 | task 0 |        eval time =   44348.00 ms /   139 tokens (  319.05 ms per token,     3.13 tokens per second)

Not sure if that's expected or not.

Some screenshots during inference:

EDIT: On a 5k token prompt I get slightly better overall performance:

65.21.916.672 I slot print_timing: id  3 | task 830 | prompt eval time =  342915.76 ms /  5038 tokens (   68.07 ms per token,    14.69 tokens per second)
65.21.916.677 I slot print_timing: id  3 | task 830 |        eval time = 1961546.27 ms /  9984 tokens (  196.47 ms per token,     5.09 tokens per second)

Also, system RAM utilization rises to the expected levels:

I think it's a little too slow to benchmark using the Aider Polyglot on this hardware, but my initial impression is that this is a pretty smart model, even at Q2.

[usrlocalben]

@createthis I observe the same wrt. warmup. Warmup should avoid top-k for _exps selection and does not, so it only loads 16/896. Without this, it takes a very long time to warmup due to coupon-collector problem. It's additionally important since as I understand, this model made efforts to improve exp distribution and avoid hot spots (improving expert-parallel workload distrib., and probably much to the dismay of the REAP-enjoyers)

[pwilkin]

@usrlocalben I can confirm, I measured possible REAP/REAM opportunities and my Opus was like "nah, better quant with a codebook instead". Very informationally dense. That's why it doesn't quant so well.

[ngxson]

IIRC, the build_* should always cont the result before returning, right?

[fairydreaming]

I'm not familiar with the gated delta net details, but from what I see in code the build_delta_net() output is permuted in some cases, so needs cont before reshape.

[fairydreaming]

Reconverted and requantized model after yesterdays changes, ran perplexity test with 8k ubatch to speed things up, the result:

$ ./bin/llama-perplexity -m /mnt/md0/models/Kimi-K3-Q2_K.gguf -f ../../perplexity/wikitext-2-raw/wiki.test.raw -c 8192 -b 8192 -ub 8192 -fit off -fa 1 -cmoe
0.00.518.407 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.565.491 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
6.38.735.162 I 
6.38.748.749 I system_info: n_threads = 32 (n_threads_batch = 32) / 64 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
6.38.748.767 I perplexity: tokenizing the input ..
6.39.145.729 I perplexity: tokenization took 396.954 ms
6.39.145.871 I perplexity: calculating perplexity over 35 chunks, n_ctx=8192, batch_size=8192, n_seq=1
8.57.891.641 I perplexity: 138.75 seconds per pass - ETA 1 hours 20.93 minutes
[1]1.2461,[2]1.2755,[3]1.5372,[4]1.4771,[5]1.4036,[6]1.3488,[7]1.5159,[8]1.4724,[9]1.4909,[10]1.4524,[11]1.5001,[12]1.6394,[13]1.6837,[14]1.7142,[15]1.6873,[16]1.6580,[17]1.6305,[18]1.6659,[19]1.6537,[20]1.6272,[21]1.6147,[22]1.6078,[23]1.5833,[24]1.5634,[25]1.5451,[26]1.5276,[27]1.5131,[28]1.5186,[29]1.5358,[30]1.5353,[31]1.5718,[32]1.5938,[33]1.5789,[34]1.5658,[35]1.5499,
43.41.992.577 I Final estimate: PPL = 1.5499 +/- 0.00478

Nice.

[fairydreaming]

Did some limited multi-stream lineage-bench runs:

• 8 lineage-64 quizzes - all passed,
• 4 lineage-128 quizzes - all passed,
• 4 lineage-256 quizzes - all passed.

[fairydreaming]

Why there are Japanese comments in the chat template?

[ngxson]

lmao, it's hilarious that there is (almost) no mentions to minja in the llama.cpp code base anymore, yet models are trained on old dataset still assumes that llama.cpp uses minja

[pwilkin]

@fairydreaming guess that's the person who posted the parser changes to my branch, I didn't check the template 😀