Server Parameters
This page documents the parameters operators usually set directly. TokenSpeed uses familiar serving parameter names where the semantics match and keeps TokenSpeed-specific knobs for runtime features with different meaning.
For a compact compatibility table, see Compatible Parameters.
Model Loading
| Parameter | Purpose |
|---|---|
positional model | Model path or Hugging Face repo ID. |
--model | Equivalent to positional model. |
--tokenizer | Tokenizer path when it differs from the model path. |
--tokenizer-mode | Select tokenizer behavior. auto uses fast tokenizers and model-specific hooks when available. |
--skip-tokenizer-init | Skip tokenizer initialization for input-ID-only serving paths. |
--load-format | Weight loading format: auto, pt, safetensors, npcache, dummy, or extensible. |
--trust-remote-code | Allow custom model code from the model repository. |
--revision | Model branch, tag, or commit. |
--download-dir | Hugging Face download/cache directory. |
--hf-overrides | JSON overrides for model configuration values. |
Precision And Quantization
| Parameter | Purpose |
|---|---|
--dtype | Model weight and activation dtype. auto follows model metadata. |
--kv-cache-dtype | KV cache dtype. Lower precision reduces KV memory and may require scaling factors. |
--kv-cache-quant-method | KV cache quantization method. |
--quantization | Weight quantization mode such as fp8, nvfp4, w8a8_fp8, or compressed-tensors. |
--quantization-param-path | JSON file for KV cache scaling factors, commonly needed with FP8 KV cache. |
API Surface
| Parameter | Purpose |
|---|---|
--host | HTTP bind host. |
--port | HTTP bind port. |
--served-model-name | Model name returned by the OpenAI-compatible API. |
--api-key | API key required by the server. |
--chat-template | Built-in chat template name or template file path (handled by the smg gateway). |
--stream-interval | Streaming buffer interval in generated tokens. Smaller values stream more frequently. |
--stream-output | Return generated text as disjoint streaming segments. |
--weight-version | Initial model-weight version stamped into generation metadata. Defaults to default. |
Weight Version Metadata
Every generation response includes the current version in meta_info["weight_version"]. RL trainers can use this value to identify the policy version that produced a sample.
The RL control plane updates the version only when the trainer supplies one:
- SGLang-compatible
update_weights_from_distributed,update_weights_from_tensor, andupdate_weights_from_diskrequests accept an optionalweight_version. The version changes only after the update succeeds. - The vLLM-compatible
finish_weight_updaterequest accepts an optionalweight_version. A version sent with an update chunk is deferred untilfinish_weight_update, so partially updated weights never advertise the new version. - Omitting
weight_versionpreserves the current value.
Use GET /get_weight_version to read the current value, POST /update_weight_version with {"new_version": "..."} to set it directly, and GET /model_info to read the model path and version together.
Scheduler And Memory
| Parameter | Purpose |
|---|---|
--max-model-len | Maximum sequence length. If omitted, TokenSpeed uses the model config. |
--gpu-memory-utilization | Fraction of GPU memory used for model weights and KV cache. Lower it to leave headroom. |
--max-num-seqs | Maximum number of active sequences the scheduler may process concurrently. |
--chunked-prefill-size | Token budget the scheduler may issue in one iteration. Defaults to 8192. Set -1 to disable chunked prefill. |
--max-prefill-tokens | Prefill token budget used when chunked prefill is disabled. Defaults to 8192. |
--max-total-tokens | Override the automatically calculated token pool size. |
--block-size | KV cache block size. |
--enable-prefix-caching / --no-enable-prefix-caching | Enable or disable prefix cache reuse. |
--enforce-eager | Disable CUDA graph execution. |
--max-cudagraph-capture-size | Largest batch size to capture with CUDA graphs. |
--cudagraph-capture-sizes | Explicit CUDA graph capture sizes. |
--chunked-prefill-size is intentionally separate from --max-num-batched-tokens: in TokenSpeed it is the scheduler's per-iteration issue budget, while --max-total-tokens controls the global token pool.
Parallelism
| Parameter | Purpose |
|---|---|
--tensor-parallel-size, --tp | Familiar alias for setting attention tensor parallel size. |
--attn-tp-size | Tensor parallel size for attention. |
--dense-tp-size | Tensor parallel size for dense layers. |
--moe-tp-size | Tensor parallel size for MoE layers. |
--data-parallel-size | Number of data-parallel replicas. |
--mm-encoder-tp-mode | Multimodal encoder parallelism: weights shards encoder weights with attention TP; data uses TP1 whole-item DP and currently requires aggregate serving, one node, and no attention context parallelism. |
--enable-expert-parallel | Set expert parallelism across the selected world size. |
--expert-parallel-size, --ep-size | Explicit expert parallel size. |
--world-size | Total worker process count across all nodes. |
--nprocs-per-node | Worker process count per node. |
--nnodes | Number of nodes. |
--node-rank | Rank of the current node. |
--dist-init-addr | Distributed initialization address. |
Use --tensor-parallel-size for simple launches. Use the TokenSpeed-specific split knobs when attention, dense, and MoE layers need different process groups.
Backend Selection
| Parameter | Purpose |
|---|---|
--attention-backend | Attention kernel backend. Common values include mha, fa3, fa4, triton, flashinfer, trtllm_mla, and tokenspeed_mla. |
--drafter-attention-backend | Attention backend for speculative decoding drafter model. |
--moe-backend | MoE backend. |
--draft-moe-backend | MoE backend for the speculative decoding draft model. |
--all2all-backend | MoE all-to-all backend. |
--deepep-mode | DeepEP mode: auto, normal, or low_latency. |
--sampling-backend | Sampling backend: greedy, flashinfer, flashinfer_full, triton, or triton_full. |
Set backend choices explicitly in production. auto is useful for bring-up, but explicit values make benchmark comparisons and regressions easier to reason about.
When --dp-sampling is enabled, the logits processor owns the per-forward logits layout decision and carries the resulting plan to the sampling backend with the logits output.
Reasoning And Tool Calling
| Parameter | Purpose |
|---|---|
--reasoning-parser | Parser for extracting reasoning content from model outputs (handled by the smg gateway). |
--tool-call-parser | Parser for OpenAI-compatible tool-call payloads (handled by the smg gateway). |
--enable-custom-logit-processor | Allow custom logit processors. Keep disabled unless the deployment needs it. |
Common reasoning parser values include kimi_k25, base, qwen3, deepseek_r1, and deepseek_v31. Common tool-call parser values include kimik2, qwen, deepseek_v4, json, and passthrough. The parser names are validated by the SMG gateway, so use the values accepted by the bundled tokenspeed-smg package.
Speculative Decoding
| Parameter | Purpose |
|---|---|
--speculative-config | JSON speculative decoding configuration. |
--speculative-algorithm | Speculative algorithm, such as EAGLE3, MTP, or DFLASH. |
--speculative-draft-model-path | Draft model path or repo ID. |
--speculative-draft-model-quantization | Draft model quantization. Defaults to unquant. |
--speculative-num-steps | Number of draft model steps. Defaults to 3. |
--speculative-num-draft-tokens | Number of draft tokens. Defaults to --speculative-num-steps + 1. |
--speculative-eagle-topk | EAGLE top-k. Defaults to 1. |
--eagle3-layers-to-capture | EAGLE3 layers to capture. |
Prefer --speculative-config for recipe-style launches because it keeps method, draft model, and token count together.
Observability
| Parameter | Purpose |
|---|---|
--log-level | Runtime log level. |
--log-level-http | HTTP server log level. Defaults to --log-level when unset. |
--enable-log-requests | Log request metadata and optionally payloads. |
--log-requests-level | Request logging verbosity. |
--enable-log-request-stats | Log a one-line per-request performance summary on finish/abort (see below). |
--enable-metrics | Enable metrics reporting. |
--metrics-reporters | Metrics reporter, such as prometheus. |
--decode-log-interval | Decode batch log interval. |
--enable-cache-report | Include cached-token counts in OpenAI-compatible usage details. |
--kv-events-config | JSON config for KV cache mutation events. Set enable_kv_cache_events and a publisher such as zmq to publish device prefix-cache stores and removals. |
Per-Request Stats
--enable-log-request-stats enriches the scheduler's per-request finish line for latency/throughput debugging. When set, the Req: <rid> Finish! ... line carries a Python-object repr (RequestStats(...)) instead of the default Accept_num_tokens_avg value (which it subsumes as acc_len). Every field is derived from host-side timestamps and counters already available in the scheduler — it adds no GPU sync and so no engine slowdown. Example:
Req: chatcmpl-019ef6b7 Finish! RequestStats(status='finished', reason='stop', prompt_tokens=28684, cache_tokens=832, output_tokens=33, cache_hit_rate=0.029, queue_ms=13.8, prefill_ms=15.8, ttft_ms=42.1, total_ms=58.0, preempt_ms=0.0, preempt_count=0, decode_tps=210.4, acc_len=None, acc_rate=None, recv_ts=1782255696.726, commit_ts=1782255696.74, finish_ts=1782255696.784)| Field | Meaning |
|---|---|
status / reason | finished vs aborted; finish-reason type (stop/length/abort). |
prompt_tokens / cache_tokens / output_tokens | Prompt tokens, prefix-cache-hit tokens, generated tokens. |
cache_hit_rate | cache_tokens / prompt_tokens (0–1). |
queue_ms | Received → first scheduled into a forward batch. |
prefill_ms | Scheduled → prefill complete. |
ttft_ms | Received → first output token (always ≥ prefill_ms; it also spans the queue). |
total_ms | Received → finished/aborted. |
preempt_ms / preempt_count | Wall-clock this request's decode was delayed by prefilling other requests, and the number of such interruptions. Host-side best-effort. |
decode_tps | Decode throughput (generated tokens / decode window). |
acc_len / acc_rate | Spec-decode acceptance length and rate (None when speculative decoding is off). |
recv_ts / commit_ts / finish_ts | Absolute epoch timestamps for received / scheduled / finished. |
KV Cache Events
KV cache events publish reusable device prefix-cache mutations from the live C++ scheduler path. Host/L2 loadback events are not published by this initial stream. Block hash lineage is cached on prefix-cache nodes, so publishing a stored block uses the parent node's cached hash instead of rebuilding the full ancestor prefix.
Example:
--kv-events-config '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5557","topic":"kv-events"}'The ZMQ publisher sends three frames: topic bytes, an 8-byte big-endian sequence number, and a msgpack payload. The payload is an array-like KVEventBatch:
[timestamp, [["BlockStored", [block_hash], parent_hash, token_ids, block_size]], attn_dp_rank]
[timestamp, [["BlockRemoved", [block_hash]]], attn_dp_rank]With attention data parallelism, each attention DP rank publishes on an offset port from the configured endpoint.
TokenSpeed-Specific Runtime Knobs
These parameters are TokenSpeed-specific. They expose runtime features directly:
--max-total-tokens--max-prefill-tokens--chunked-prefill-size--attn-tp-size--dense-tp-size--moe-tp-size--kvstore-*--enable-mla-l1-5-cache--kv-events-config--mla-chunk-multiplier--disaggregation-*--comm-fusion-max-num-tokens--enable-allreduce-fusion