max serve

max serve​

Usage​

max serve [OPTIONS]

Options​

• --allow-cold-interpreter-cache​

  Permit compiling eager interpreter ops on demand, instead of refusing on a machine that max warm-interpreter-cache has not warmed. Equivalent to setting MAX_EAGER_ALLOW_LAZY_COMPILE=1.

  Default:

  False

• --allow-extra-request-fields, --no-allow-extra-request-fields​

  When True, unknown top-level fields on OpenAI-compatible request bodies are dropped with a warning before pydantic validation, instead of producing a 400.

• --allow-unsupported-logprobs, --no-allow-unsupported-logprobs​

  When True, OpenAI-compatible requests that ask for logprobs against a runtime configuration that cannot honor them will raise a warning, and served as if logprobs were not requested. Each response chunk carries logprobs: null. When False (default), such requests are rejected with a 400.

• --cascade, --no-cascade​

  Serve via the experimental Cascade server (max.experimental.cascade.serve.main.serve) instead of the standard API server + model worker. The resolved PipelineArgs is forwarded to the Cascade entrypoint.

• --cascade-local-cpu-workers <cascade_local_cpu_workers>​

  [--cascade] Number of local CPU worker subprocesses. Defaults to 2.

• --cascade-local-gpu-workers <cascade_local_gpu_workers>​

  [--cascade] Number of local GPU worker subprocesses. When unset, auto-sized from --devices (one per distinct GPU set).

• --cascade-remote-cpu-workers <cascade_remote_cpu_workers>​

  [--cascade] Addresses of already-running remote CPU workers to include in the pool (repeatable). Format depends on --cascade-transport: an http:// or unix:// URL for http, host:port (or grpc://host:port) for grpc.

• --cascade-remote-gpu-workers <cascade_remote_gpu_workers>​

  [--cascade] Addresses of already-running remote GPU workers to include in the pool (repeatable). Format depends on --cascade-transport: an http:// or unix:// URL for http, host:port (or grpc://host:port) for grpc.

• --cascade-transport <cascade_transport>​

  [--cascade] Worker transport: http (default) or grpc. Sets the address format expected by --cascade-remote-cpu-workers / --cascade-remote-gpu-workers.

  Options:

  http | grpc

• --ce-delay-ms <ce_delay_ms>​

  Duration of scheduler sleep prior to starting a prefill batch.

• --chat-template <chat_template>​

  Path to a custom chat template that overrides the one from the Hugging Face model (the default). The file is either a Jinja template or a JSON object with a chat_template key.

• --chunked-prefill-min-chunk-size <chunked_prefill_min_chunk_size>​

  Floor, in tokens, on any chunk created by chunked prefill. When splitting a request against the CE token budget, the cut is moved earlier so that neither the chunk nor the remainder is smaller than this; if no legal cut point exists within the remaining budget, the request is left unsplit for a later step. 0 (default) disables the floor: cuts land exactly on the budget boundary, which can produce very small chunks. Values above max_batch_input_tokens / 2 forbid most splits; a sane range is roughly 64-1024.

• --config-file <config_file>​

• --custom-architectures <custom_architectures>​

  Custom architecture implementations to register. Each input is either a path to a single custom-architecture module directory or an IMPORT_PATH:MODULE_NAME colon-form. Each module must expose a top-level ARCHITECTURES list of SupportedArchitecture instances.

• --data-parallel-degree <data_parallel_degree>​

  Data-parallelism parameter. The degree to which the model is replicated is dependent on the model type.

• --debug-verify-replay, --no-debug-verify-replay​

  When device_graph_capture is enabled, execute eager launch-trace verification before replay. Intended for debugging only.

• --decode-request-ttl-s <decode_request_ttl_s>​

  Per-request TTL in seconds for the decode-side prefill_reqs and inflight_transfers dicts. Entries older than this are evicted individually (KV blocks released, failure surfaced to the client) before the stall watchdog fires. None (the default) disables eviction. Set with the MODULAR_DECODE_REQUEST_TTL_S environment variable.

• --decode-stall-timeout-s <decode_stall_timeout_s>​

  Seconds of no-batch-activity after which the decode worker exits to trigger a pod restart. None (the default) disables the watchdog. Set with the MODULAR_DECODE_STALL_TIMEOUT_S environment variable.

• --device-graph-capture, --no-device-graph-capture​

  Enable device graph capture and replay for graph execution. If unset, automatically enabled for some selected architectures. Use --no-device-graph-capture to explicitly disable.

• --device-memory-utilization <device_memory_utilization>​

  The fraction of available device memory that the process should consume. The remaining headroom holds the KV cache: kv_cache_workspace = (total_free_memory * device_memory_utilization) - model_weights_size.

• --devices <devices>​

  Whether to run the model on CPU (--devices=cpu), GPU (--devices=gpu), every visible GPU (--devices=gpu:all), or a list of GPUs (--devices=gpu:0,1). An ID value can be provided optionally to indicate the device ID to target. If not provided, the model or config default is used.

• --dp-ce-balance-enable-dynamic-chunk-size, --no-dp-ce-balance-enable-dynamic-chunk-size​

  Whether a below-threshold CE step with work on 2+ replicas runs immediately with each replica's chunk size reduced to the balance level, deferring only the excess. When False, such steps are held whole until the threshold is met, a deadline expires, or there is nothing else to run. Only consulted when dp_ce_balance_timeout_ms > 0.

• --dp-ce-balance-threshold <dp_ce_balance_threshold>​

  Per-step CE active-token occupancy across DP replicas (mean/max, 0-1) at or above which CE work is scheduled without further deferral. Only consulted when dp_ce_balance_timeout_ms > 0.

• --dp-ce-balance-timeout-ms <dp_ce_balance_timeout_ms>​

  Max time in milliseconds a context-encoding request's work may be deferred, from arrival, while awaiting token-balanced scheduling across data-parallel replicas. -1 disables the balancer (requests bind to a replica on arrival; current default behavior); 0 enables post-cache-weighted placement with late binding but never defers; > 0 additionally defers unbalanced CE work until dp_ce_balance_threshold is met, the deadline expires, or there is nothing else to run.

• --draft-chat-template <draft_chat_template>​

  Path to a custom chat template that overrides the one from the Hugging Face model (the default). The file is either a Jinja template or a JSON object with a chat_template key.

• --draft-config-file <draft_config_file>​

• --draft-data-parallel-degree <draft_data_parallel_degree>​

  Data-parallelism parameter. The degree to which the model is replicated is dependent on the model type.

• --draft-devices <draft_devices>​

  Devices for the draft model in speculative decoding. If not provided, inherits from --devices. Accepts the same format as --devices.

• --draft-enable-echo, --no-draft-enable-echo​

  Whether the model should be built with echo capabilities.

• --draft-force-download, --no-draft-force-download​

  Whether to force download a given file if it's already present in the local cache.

• --draft-huggingface-model-revision <draft_huggingface_model_revision>​

  Branch or Git revision of Hugging Face model repository to use.

• --draft-huggingface-weight-revision <draft_huggingface_weight_revision>​

  Branch or Git revision of Hugging Face model repository to use.

• --draft-max-length <draft_max_length>​

  Maximum sequence length the model can process. If not specified, defaults to the model's max_position_embeddings. Resolved to the architecture's policy value at construction; memory planning may lower it for VRAM on the memory plan only, never here.

• --draft-model-path <draft_model_path>​

  Accepts either a Hugging Face repository ID or a local path to the model.

• --draft-pool-embeddings, --no-draft-pool-embeddings​

  Whether to pool embedding outputs.

• --draft-proposal <draft_proposal>​

  How the draft model proposes tokens. 'argmax' (default) proposes deterministically. 'sampled' makes the draft sample its own proposal and keep the distribution it drew from, so verification runs true speculative sampling instead of typical acceptance. Incompatible with use_relaxed_acceptance_for_thinking. Inert unless the serving architecture supports it.

  Options:

  argmax | sampled

• --draft-quantization-encoding <draft_quantization_encoding>​

  Weight encoding type. For GGUF models, the encoding is auto-detected from the repository when unset; if set, it must match an available encoding. When the repository contains multiple quantization formats, set this to choose one.

  Options:

  float32 | float16 | bfloat16 | q4_k | q4_0 | q6_k | float8_e4m3fn | float4_e2m1fnx2 | float6_e2m3fn | gptq

• --draft-rope-type <draft_rope_type>​

  Force using a specific rope type. Only matters for GGUF weights.

  Options:

  none | normal | neox | longrope | yarn

• --draft-section-name <draft_section_name>​

• --draft-served-model-name <draft_served_model_name>​

  Optional override for client-facing model name. Defaults to model_path.

• --draft-sliding-window <draft_sliding_window>​

  If set, overrides the model's attention to use a sliding-window causal mask of this many tokens. None (the default) defers to the HuggingFace config's sliding_window field, or full causal attention if the model doesn't advertise one.

• --draft-subfolder <draft_subfolder>​

  Subdirectory within the HuggingFace repo to load config and weights from (for example, vae or text_encoder). When set, config.json and weights are resolved from {model_path}/{subfolder}/.

• --draft-trust-remote-code, --no-draft-trust-remote-code​

  Whether or not to allow for custom modeling files on Hugging Face.

• --draft-use-subgraphs, --no-draft-use-subgraphs​

  Whether to use subgraphs for the model. This can significantly reduce compile time, especially for large models with identical blocks. Default is true.

• --draft-vision-config-overrides <draft_vision_config_overrides>​

  Model-specific vision configuration overrides. For example, for InternVL: {"max_dynamic_patch": 24}.

• --draft-weight-path <draft_weight_path>​

  Optional path or URL of the model weights to use. Overrides default weight discovery.

• --eager-usage-validator <eager_usage_validator>​

  Configure when to use the eager usage validator, which reports unsafe API usage. init-only (default) validates model initialization only; enabled validates both model initialization and every request, at the cost of Python overhead on the serving path; disabled turns it off.

  Options:

  enabled | disabled | init-only

• --emit-reasoning-content, --no-emit-reasoning-content​

  When True, chat completion responses emit a thinking model's chain-of-thought under reasoning_content only (reasoning is omitted). The reasoning_content alias is used by vLLM, SGLang, and the DeepSeek API; some clients require it. When False (default), responses emit reasoning under reasoning only.

• --enable-chunked-prefill, --no-enable-chunked-prefill​

  Enable chunked prefill to split context encoding requests into multiple chunks based on max_batch_input_tokens.

• --enable-dp-cross-replica-prefix-copy, --no-enable-dp-cross-replica-prefix-copy​

  Whether a prefix-cache block resident on another data-parallel (DP) replica's GPU may be copied device-to-device onto the request's replica to serve a cache hit. When disabled, cross-replica reuse is only served from the shared host/disk tier via the KV connector (or recomputed). Only relevant when data_parallel_degree > 1 and prefix caching is enabled.

• --enable-echo, --no-enable-echo​

  Whether the model should be built with echo capabilities.

• --enable-in-flight-batching, --no-enable-in-flight-batching​

  When enabled, prioritizes token generation by batching it with context encoding requests.

• --enable-lora, --no-enable-lora​

  Enables LoRA on the server.

• --enable-min-tokens, --no-enable-min-tokens​

  Whether to enable min_tokens, which blocks the model from generating stopping tokens before the min_tokens count is reached.

• --enable-overlap-scheduler, --no-enable-overlap-scheduler​

  Whether to enable the overlap scheduler. This feature allows the scheduler to run alongside GPU execution. This helps improve GPU utilization. This is an experimental feature which may crash and burn. This feature will be enabled by default for some selected architectures. You can forcibly disable this by setting --no-enable-overlap-scheduler --force.

• --enable-penalties, --no-enable-penalties​

  Whether to apply frequency and presence penalties to the model's output.

• --enable-prefix-caching, --no-enable-prefix-caching​

  Whether to enable prefix caching for the paged KVCache.

• --enable-prioritize-first-decode, --no-enable-prioritize-first-decode​

  When enabled, the scheduler always runs a TG batch immediately after a CE batch with the same requests. This may reduce time-to-first-chunk latency.

• --enable-spec-decode-mixed-batches, --no-enable-spec-decode-mixed-batches​

  When enabled with speculative decoding, prefill requests are batched into decode steps (implies in-flight batching) and the decode rows keep verifying their draft tokens instead of advancing draft-less. Architectures that do not declare supports_spec_decode_mixed_batches fall back to plain in-flight batching.

• --enable-structured-output, --no-enable-structured-output​

  Enable structured generation/guided decoding for the server. This allows the user to pass a JSON schema in the response_format field, which the LLM will adhere to.

• --enable-tool-call-constrained-decode, --no-enable-tool-call-constrained-decode​

  Whether tool-call requests are constrained to a server-generated grammar during decoding. When enabled (the default), a configured runtime.tool_parser both produces a decode-time grammar and parses the resulting output. Set to False to keep the parser (tool calls are still parsed out of generated text) while skipping the constrained-decode/bitmask path for tool calls -- useful when the grammar path is undesirable but tool-call parsing is still wanted. With this disabled, tool_choice=required or a named function can no longer force a tool call. Independent of enable_structured_output, which gates user-supplied response_format JSON schemas.

• --enable-variable-logits, --no-enable-variable-logits​

  Enable the sampling graph to accept a ragged tensor of different sequences as inputs, along with their associated logit_offsets. This is needed to produce additional logits for echo and speculative decoding purposes.

• --ep-fuse-ffn-combine-send, --no-ep-fuse-ffn-combine-send​

  Whether the MoE FFN performs the expert-parallel combine send from its own epilogue, which drops the local staging tensor for its output. Unset defaults ON except for a decode-only worker, since the span it speeds up grows with tokens per expert.

• --ep-size <ep_size>​

  The expert parallelism size. Needs to be 1 (no expert parallelism) or the total number of GPUs across nodes.

• --ep-use-allreduce, --no-ep-use-allreduce​

  Whether to use allreduce for the cross-device communication in expert parallelism.

• --eplb-profile, --no-eplb-profile​

  When True, enables expert-parallel load balancing (EPLB) MoE routing histogram profiling in the pipeline. Mirrors Settings.eplb_profile for pipeline code that doesn't have access to Settings.

• --eplb-replicas-per-gpu <eplb_replicas_per_gpu>​

  Number of redundant expert replicas to add per GPU when EPLB is active. 0 (default) means no replication. k > 0 adds k extras per GPU; total redundant slots = k * ep_size (so num_redundant is always a multiple of the device count, which the rebalance algorithm requires).

• --eplb-stats <eplb_stats>​

  Path to a snapshot JSON from /max_internal/eplb_stats. Triggers an EPLB rebalance at startup.

• --execute-empty-batches, --no-execute-empty-batches​

  When enabled, the scheduler runs the model's forward pass even for an empty batch, so expert-parallel and data-parallel replicas still reach their collective barrier points; output processing is skipped. The architecture must support empty batches.

• --experimental-device-graph-synthesis, --no-experimental-device-graph-synthesis​

  Compile model graphs with device-graph synthesis: the compiled model constructs a device graph directly and executes it on model forward passes. This is an experimental alternative to the capture/replay workflow. Honored only by architectures that opt in, and mutually exclusive with device_graph_capture. Use --experimental-device-graph-synthesis to enable.

• --export-mefs <export_mefs>​

  Directory to write a compiled-graph artifact into for every graph this run compiles, for a later run to reuse via --precompiled-mefs. Compilation itself is unaffected.

• --first-block-caching, --no-first-block-caching​

  Enable First-Block Cache (FBCache) for step-cache denoising. When enabled, the transformer skips remaining blocks if the first-block residual is similar to the previous step.

• --force, --no-force​

  Skip validation of user provided flags against the architecture's required arguments.

• --force-download, --no-force-download​

  Whether to force download a given file if it's already present in the local cache.

• --gpu-profiling <gpu_profiling>​

  Whether to enable GPU profiling of the model.

  Options:

  off | on | detailed

• --headless​

  Run only the dispatcher service and model worker without the API server.

  Default:

  False

• --host <host>​

  Interface to bind the HTTP API to. Defaults to MAX_SERVE_HOST.

• --huggingface-model-revision <huggingface_model_revision>​

  Branch or Git revision of Hugging Face model repository to use.

• --huggingface-weight-revision <huggingface_weight_revision>​

  Branch or Git revision of Hugging Face model repository to use.

• --indexer-kv-cache-format <indexer_kv_cache_format>​

  Override the MiniMax sparse-indexer (IndexK) cache dtype, independent of kv_cache_format. Supported values: bfloat16, float8_e4m3fn. None (default) keeps IndexK in bfloat16 so --kv-cache-format=float8_e4m3fn still means main GQA FP8 plus indexer BF16. Ignored by architectures without an indexer cache. FP8 IndexK is scale-free and AMD-only.

• --kv-cache-format <kv_cache_format>​

  Override the default data type for the KV cache. Supported values: float32, bfloat16, float8_e4m3fn.

• --kv-cache-hash-algo <kv_cache_hash_algo>​

  Hash algorithm used for KV-cache block identity. ahash64 (default) is fast and non-cryptographic; sha256 is a cryptographic 256-bit hasher; both support an optional seed/salt for prefix-cache isolation. sha256_64 truncates the SHA-256 chain to 64 bits for protocol compatibility.

  Options:

  ahash64 | sha256 | sha256_64

• --kv-cache-hash-seed <kv_cache_hash_seed>​

  Optional 64-character hex string (32 bytes), a cluster-wide seed for kv_cache_hash_algo. If omitted, sha256/sha256_64 generate a random seed at startup; ahash64 does not, so existing deployments are unaffected unless set explicitly.

• --kv-cache-page-size <kv_cache_page_size>​

  The number of tokens in a single page in the paged KVCache.

• --kv-connector-config <kv_connector_config>​

  KV cache connector configuration as inline JSON or a path to a YAML/JSON file. The connector type is the type field, e.g. '{"type": "rust_tiered"}'. Defaults to the null connector (no external caching); each type has sensible defaults for its remaining fields. Merges field-wise over a config file's value, so overriding one field on the command line preserves the rest.

• --log-prefix <log_prefix>​

  Optional prefix to add to all log messages for this server instance.

• --lora-paths <lora_paths>​

  List of statically defined LoRA paths.

• --max-batch-input-tokens <max_batch_input_tokens>​

  The target number of un-encoded tokens to include in each batch. This value is used for chunked prefill and memory estimation.

• --max-batch-size <max_batch_size>​

  Maximum batch size to execute with the model. When not specified (None), this value is determined dynamically. For server launches, set this higher based on server capacity.

• --max-batch-total-tokens <max_batch_total_tokens>​

  Ensures the sum of page-aligned context lengths in a batch does not exceed max_batch_total_tokens. Alignment uses the KV cache page size. If None, the sum is not limited.

• --max-length <max_length>​

  Maximum sequence length the model can process. If not specified, defaults to the model's max_position_embeddings. Resolved to the architecture's policy value at construction; memory planning may lower it for VRAM on the memory plan only, never here.

• --max-lora-rank <max_lora_rank>​

  Maximum rank of all possible LoRAs.

• --max-media-preprocess-cache-idle-seconds <max_media_preprocess_cache_idle_seconds>​

  How long a preprocessed image or video may go unused before it becomes eligible to be dropped from the tokenizer's cache. This is a reclaim policy rather than a lifetime: sweeps are periodic, so an entry can outlive its deadline, and a request that arrives meanwhile is served from it and resets the clock -- an entry is keyed on media content, so it never goes stale. Without this, the byte budget is the only bound, so a burst of distinct media holds its whole resident set for the rest of the process's life -- host memory the model worker's own allocations compete for. An entry is only worth keeping while the conversation that sent it might send the next turn, which is seconds to minutes, and re-preprocessing a wrongly dropped image costs a few milliseconds. Set to 0 to keep entries until the budget evicts them. Only used by VLMs.

• --max-num-loras <max_num_loras>​

  The maximum number of active LoRAs in a batch. This controls how many LoRA adapters can be active simultaneously during inference. Lower values reduce memory usage but limit concurrent adapter usage.

• --max-pending-requests <max_pending_requests>​

  Cap (M) on the scheduler's pending (prefill) queue depth. The worker stops pulling from the request queue once it holds this many not-yet-running requests, so the request queue backs up and exerts backpressure (see --max-queue-size) instead of growing an unbounded pending pool. Should be at least --max-batch-size. Defaults to unbounded.

• --max-queue-size <max_queue_size>​

  Cap (N) on the request queue to the model worker. Once this many requests are in transit to the worker, new requests are rejected with HTTP 429 instead of being enqueued, providing self-calibrating backpressure to keep latency within SLAs. Pair with --max-pending-requests. Defaults to unbounded.

• --max-queue-size-tg <max_queue_size_tg>​

  Maximum number of requests in decode queue. By default, this is max_batch_size.

• --max-request-input-tokens <max_request_input_tokens>​

  Ceiling, in tokens, on how much of a single request is encoded in one batch, so short requests still fit alongside a long prefill instead of queueing behind it. 0 (default) disables the cap. Requires chunked prefill.

• --max-video-preprocess-cache-bytes <max_video_preprocess_cache_bytes>​

  Host-memory budget, in bytes, for caching preprocessed video tensors in the tokenizer. Unlike images, videos are not decoded at admission, so a hit skips the whole decode -- sampling, resize and patchify of every sampled frame. Budgeted separately from max_vision_preprocess_cache_bytes because a video entry is an order of magnitude larger than an image one, so a shared budget would let a single video evict many images. Set to 0 to disable. Only used by VLMs that accept video.

• --max-vision-preprocess-cache-bytes <max_vision_preprocess_cache_bytes>​

  Host-memory budget, in bytes, for caching preprocessed image tensors in the tokenizer. A hit skips the resize, rescale and patchify for a repeated image -- for example the same image resent on every turn of a conversation -- which the vision encoder cache cannot avoid, because it is consulted only after preprocessing has already run. This is a ceiling on resident host memory in the API server process, not a reservation: the cache grows to it under load and evicts least-recently-used entries to stay within it. Set to 0 to disable. Only used by VLMs.

• --min-batch-size-tg <min_batch_size_tg>​

  Soft floor on the decode batch size. If the TG batch size is larger, the scheduler continues TG batches; if it falls below, the scheduler prioritizes CE. This is not a strict minimum. By default, this is max_queue_size_tg.

• --model, --model-path <model_path>​

  Accepts either a Hugging Face repository ID or a local path to the model.

• --model-override <model_override>​

  Per-component overrides for the ModelManifest, in the format component.field=value. Applied before resolution. Repeatable. Example: transformer.quantization_encoding=float4_e2m1fnx2.

• --models <models>​

  The model manifest containing all model configs keyed by role.

• --num-speculative-tokens <num_speculative_tokens>​

  The number of speculative tokens. Unset selects a per-method default: 2 for eagle/mtp, and the draft checkpoint's trained width for dflash.

• --num-speculative-tokens-mixed-batch <num_speculative_tokens_mixed_batch>​

  How many drafted tokens the target verifies on a mixed prefill+decode batch. Unset uses the count a pure decode batch of the same size would.

• --num-speculative-tokens-per-batch-size <num_speculative_tokens_per_batch_size>​

  Batch-size schedule for how many drafted tokens to verify, as inclusive ranges. For example '[{"batch_start": 1, "batch_end": 16, "num_tokens": 3}, {"batch_start": 17, "batch_end": 64, "num_tokens": 1}]'. Unset verifies every drafted token.

• --pipeline-role <pipeline_role>​

  Whether the pipeline should serve both a prefill or decode role or both.

  Options:

  prefill_and_decode | prefill_only | decode_only

• --pool-embeddings, --no-pool-embeddings​

  Whether to pool embedding outputs.

• --port <port>​

  Port for the HTTP API. Defaults to 8000.

• --precompiled-mefs <precompiled_mefs>​

  Directory of compiled-graph artifacts written by an earlier run's --export-mefs. Every graph is initialized from its artifact instead of being compiled, so the compiling and the executing run can happen on different machines. The runs must build the same graphs; a mismatch is an error rather than a silent recompile.

• --prefer-module-v3, --no-prefer-module-v3​

  Whether to prefer the ModuleV3 architecture over the graph API architecture. When False (default), the inference server uses the graph API architecture. When True, the server uses the ModuleV3 architecture when available and falls back to the graph API architecture.

• --prefill-coalesce-max-held-steps <prefill_coalesce_max_held_steps>​

  Ceiling on how many consecutive decode steps a prefill is held before it is admitted regardless of how few are queued. This splits the release deadline out of prefill_coalesce_min_pending, which otherwise serves as both the queue depth that opens the gate and the step count that expires it. The default of 0 falls back to prefill_coalesce_min_pending.

• --prefill-coalesce-min-pending <prefill_coalesce_min_pending>​

  Under in-flight batching the scheduler holds queued fresh prefill requests until enough can share one decode step. Prefills are released once this many are queued, counted across the whole data-parallel group, since one eager step serves every replica at once. A held prefill is also released after this many decode steps so it never waits indefinitely, unless prefill_coalesce_max_held_steps sets that deadline separately. Holding keeps more steps pure decode so they can replay the captured graph. A step holding a prefill request runs eager. The default of 0 holds nothing back.

• --prefill-schedule-interval <prefill_schedule_interval>​

  Admit prefill work only on every Nth scheduler step. Data-parallel ranks advance in lockstep, so prefill on any rank stalls the whole group; confining it to a shared cadence leaves the steps in between entirely to decode. A step with no decode work anywhere admits prefill regardless, rather than run an empty batch. The default of 1 admits prefill every step.

• --pretty-print-config​

  Pretty Print Entire Config

• --quantization-encoding <quantization_encoding>​

  Weight encoding type. For GGUF models, the encoding is auto-detected from the repository when unset; if set, it must match an available encoding. When the repository contains multiple quantization formats, set this to choose one.

  Options:

  float32 | float16 | bfloat16 | q4_k | q4_0 | q6_k | float8_e4m3fn | float4_e2m1fnx2 | float6_e2m3fn | gptq

• --reasoning-parser <reasoning_parser>​

  Name of the reasoning output parser. The parser extracts thinking blocks to populate the reasoning field in chat completion responses. When unset, the server applies the architecture's default reasoning parser, if any. Pass "none" (case-insensitive) to explicitly disable reasoning parsing even when the architecture declares a default.

• --rejection-sampling-strategy <rejection_sampling_strategy>​

  Rejection sampling strategy for verifying draft tokens. Currently inert: the architecture's AcceptanceSampler decides the acceptance rule.

  Options:

  greedy | residual | typical-acceptance | logit-comparison

• --relaxed-delta <relaxed_delta>​

  Probability gap below the top-1 candidate inside which candidates remain eligible for relaxed acceptance. A draft token is accepted if it matches any top-N candidate whose probability is at least top1_prob - relaxed_delta. Ignored when use_relaxed_acceptance_for_thinking is False.

• --relaxed-topk <relaxed_topk>​

  Top-N candidates from the target distribution to consider when relaxed acceptance is active. Ignored when use_relaxed_acceptance_for_thinking is False.

• --rope-type <rope_type>​

  Force using a specific rope type. Only matters for GGUF weights.

  Options:

  none | normal | neox | longrope | yarn

• --sample-on-host, --no-sample-on-host​

  Run the token sampler on the host CPU instead of the model device. The last-token logits are copied device-to-host and sampling (top-k/argmax) runs on CPU. Default is to sample on the model device.

• --section-name <section_name>​

• --served-model-name <served_model_name>​

  Optional override for client-facing model name. Defaults to model_path.

• --sliding-window <sliding_window>​

  If set, overrides the model's attention to use a sliding-window causal mask of this many tokens. None (the default) defers to the HuggingFace config's sliding_window field, or full causal attention if the model doesn't advertise one.

• --speculative-method <speculative_method>​

  The speculative decoding method to use.

  Options:

  eagle | mtp | dflash | dflash2

• --state-pool-dtype <state_pool_dtype>​

  Override the storage dtype of a hybrid model's recurrent state pools (SSM/linear-attention conv and recurrent state). Defaults to the model's compute dtype (bfloat16 for supported architectures). float32 makes a speculated generation follow the exact state trajectory of an unspeculated one, at roughly double the per-request state memory (Qwen3.8-27B: 74.8 to 149.6 MiB per seated request). Supported values: bfloat16, float32.

• --structured-output-any-whitespace, --no-structured-output-any-whitespace​

  Whether structured-output (response_format) grammars accept whitespace between JSON tokens. False (the resolved default) constrains generation to compact JSON -- no whitespace, ','/':' separators -- which mitigates runaway generation on some models but also masks the newline/indentation tokens models prefer at structural boundaries. True uses the grammar engine's whitespace-tolerant JSON. When unset (None), resolved at config construction to the architecture's default if it declares one, else False. An explicit value always wins. Tool-call grammars are unaffected.

• --structured-output-backend <structured_output_backend>​

  Grammar backend for constrained decoding. Supported value: xgrammar. When unset (None), resolved at config construction to the architecture's default if it declares one, else the global default xgrammar. An explicit value always wins.

• --subfolder <subfolder>​

  Subdirectory within the HuggingFace repo to load config and weights from (for example, vae or text_encoder). When set, config.json and weights are resolved from {model_path}/{subfolder}/.

• --synthetic-acceptance-rate <synthetic_acceptance_rate>​

  Synthetic acceptance rate for benchmarking (0.0 to 1.0). When set, the rejection sampler bypasses the real draft/target comparison and accepts each draft position with a calibrated probability so the mean joint acceptance across num_speculative_tokens positions matches this value.

• --task <task>​

  The pipeline task to run (e.g. text_generation, embeddings_generation). Used to disambiguate architectures registered under the same name for multiple tasks.

  Options:

  PipelineTask.TEXT_GENERATION | PipelineTask.EMBEDDINGS_GENERATION | PipelineTask.PIXEL_GENERATION | PipelineTask.AUDIO_GENERATION | PipelineTask.UNDEFINED

• --task-arg <task_arg>​

  Task-specific arguments to pass to the underlying model (can be used multiple times).

• --taylorseer, --no-taylorseer​

  Enable TaylorSeer cache optimization. Uses Taylor series prediction to skip full transformer passes on certain denoising steps.

• --taylorseer-cache-interval <taylorseer_cache_interval>​

  Steps between full TaylorSeer computations. None uses the model-specific default (typically 5).

• --taylorseer-max-order <taylorseer_max_order>​

  Taylor expansion order (1 or 2). Higher order uses second derivatives for more accurate prediction. None uses the model-specific default (typically 1).

• --taylorseer-warmup-steps <taylorseer_warmup_steps>​

  Number of warmup steps before TaylorSeer prediction begins. None uses the model-specific default (typically 4).

• --temperature <temperature>​

  Default sampling temperature. Controls randomness of token selection—higher values (e.g. 1.0) produce more random outputs, lower values (e.g. 0.2) produce more deterministic outputs. When set, this server-level default applies to all requests that do not explicitly provide temperature.

• --thinking-temperature <thinking_temperature>​

  Default temperature override for tokens inside <think>...</think> blocks. When set, this server-level default applies to all requests that do not explicitly provide thinking_temperature. Requires a reasoning parser to be configured; ignored otherwise.

• --tool-parser <tool_parser>​

  Name of the tool call parser. The parser extracts tool calls from model output in chat completion responses. When unset, the server applies the architecture's default tool parser, if any. Pass "none" (case-insensitive) to explicitly disable tool parsing even when the architecture declares a default.

• --top-k <top_k>​

  Default top-k sampling limit. When set, this server-level default applies to all requests that do not explicitly provide top_k.

• --trust-remote-code, --no-trust-remote-code​

  Whether or not to allow for custom modeling files on Hugging Face.

• --use-experimental-kernels <use_experimental_kernels>​

  Enables using experimental Mojo kernels with max serve. The kernels could be unstable or incorrect.

• --use-greedy-acceptance, --no-use-greedy-acceptance​

  Use greedy (argmax) draft acceptance instead of the stochastic sampler. The greedy path has no mid-graph allocation, so the fused speculative graph can be CUDA-graph captured. Valid only for greedy serving (temperature 0, top_k 1); incompatible with relaxed and synthetic acceptance.

• --use-relaxed-acceptance-for-thinking, --no-use-relaxed-acceptance-for-thinking​

  Enables relaxed acceptance for speculative decoding draft positions inside a <think>...</think> block. The target's top-N candidates (filtered by a probability threshold top1_prob - relaxed_delta) are compared against the draft token; matching any candidate accepts the draft. Outside the thinking span, the existing strict acceptance rule still applies. Requires draft_proposal='argmax'.

• --use-subgraphs, --no-use-subgraphs​

  Whether to use subgraphs for the model. This can significantly reduce compile time, especially for large models with identical blocks. Default is true.

• --use-vendor-blas <use_vendor_blas>​

  Enables using vendor BLAS libraries (cublas, hipblas, etc.) with max serve. Currently, this just replaces matmul calls.

• --use-vendor-ccl <use_vendor_ccl>​

  Enables using vendor CCL libraries (NCCL/RCCL) for collective operations such as allreduce in multi-GPU inference.

• --vision-cache-utilization <vision_cache_utilization>​

  Fraction of the KV cache pool budget (not total device memory) reserved for the vision encoder cache, which stores per-image encoder output to avoid re-encoding across chunks and requests; the remainder stays with the KV cache. The budget is carved into fixed-size blocks (a video spans many blocks, an image a few). 0 disables caching; the default reserves a small slice of the pool. Only used by VLMs.

• --vision-config-overrides <vision_config_overrides>​

  Model-specific vision configuration overrides. For example, for InternVL: {"max_dynamic_patch": 24}.

• --weight-path <weight_path>​

  Optional path or URL of the model weights to use. Overrides default weight discovery.