Skip to content

Add GLM-5.2 low-latency serving recipe - #652

Draft
simon-mo wants to merge 1 commit into
mainfrom
agent/glm-5-2-low-latency
Draft

Add GLM-5.2 low-latency serving recipe#652
simon-mo wants to merge 1 commit into
mainfrom
agent/glm-5-2-low-latency

Conversation

@simon-mo

Copy link
Copy Markdown
Contributor

What changed

  • Pin the GLM-5.2 NVFP4 low-latency path to vllm/vllm-openai:v0.25.1.
  • Configure the single-node Blackwell command with TP8, FP8 KV cache, MTP=5, the fused GLM model-class override, breakable piecewise CUDA graphs, tool/reasoning parsers, and the stable served model name.
  • Make Docker the primary install path and replace the stale :glm52 guide example with the release-tested copy-paste command.
  • Preserve variant-forced default_modes when building the static JSON API.

Why

The public GLM-5.2 guide still showed the older image and a generic NVFP4 command, while the current low-latency path requires vLLM 0.25.1 and explicit fused-model/CUDA-graph configuration.

The API generator also excluded opt-in features unconditionally. As a result, promoted variant endpoints dropped MTP even when a variant's default_modes explicitly enabled it. The generator now mirrors the interactive command builder.

Impact

Selecting the NVFP4 variant on B200/B300, or consuming /nvidia/GLM-5.2-NVFP4.json, produces the same low-latency command with MTP=5 and the pinned Docker image.

Validation

  • node scripts/build-recipes-api.mjs
  • Directly inspected the generated native and Docker commands in public/nvidia/GLM-5.2-NVFP4.json.
  • git diff --check

pnpm lint was not available as a non-interactive check because this repository has no ESLint configuration and next lint opens the first-run configuration prompt.

Signed-off-by: simon-mo <simon.mo@hey.com>
@vercel

vercel Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
vllm-recipes Ready Ready Preview, Comment Jul 17, 2026 8:59pm

Request Review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the GLM-5.2 model recipe to support the low-latency NVFP4 variant on Blackwell GPUs. It introduces specific installation notes, prerequisites, and detailed Docker/CLI command configurations for NVFP4, including MTP speculative decoding, model class overrides, and breakable piecewise CUDA graphs. Additionally, the recipe build script (build-recipes-api.mjs) is updated to automatically enable features forced by a variant's default_modes. There are no review comments, and I have no feedback to provide.

@gaby

gaby commented Jul 22, 2026

Copy link
Copy Markdown

@simon-mo Fix the merge conflict

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants