feat(server)!: add --gpu to choose the GPU on multi-GPU machines - #117
Merged
Conversation
--gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index. It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>. /v1/stats reports the engine's GPU under "gpus". ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name. The daemon's /bench/profile returns the running serve's GPU profile. BREAKING CHANGE: ft checkpoint --device is removed; use --gpu.
…DEVICES BREAKING CHANGE: ft bench bw --device is removed; use --gpu.
Splits the published id into _assigned_physical (UUID) and _assigned_visible (CUDA ordinal). One process runs on one GPU, so e4m3_native() judges that card instead of scanning every visible device.
Collaborator
|
@jason-fxz LGTM. I think we can merge it now |
This was referenced Aug 25, 2026
Samfisheryu
pushed a commit
to Samfisheryu/FreeToken
that referenced
this pull request
Aug 26, 2026
…shML-org#117) * feat(server)!: add --gpu to choose the GPU on multi-GPU machines --gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index. It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>. /v1/stats reports the engine's GPU under "gpus". ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name. The daemon's /bench/profile returns the running serve's GPU profile. BREAKING CHANGE: ft checkpoint --device is removed; use --gpu. * refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES BREAKING CHANGE: ft bench bw --device is removed; use --gpu. * refactor: name the id namespaces and drop the multi-device e4m3 scan Splits the published id into _assigned_physical (UUID) and _assigned_visible (CUDA ordinal). One process runs on one GPU, so e4m3_native() judges that card instead of scanning every visible device. (cherry picked from commit 2757bb5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ft servealways usedcuda:0. This adds--gputo pick the card:ft bench bwandft checkpointtake the same flag. The Desktop app uses it for its GPU picker.Closes #99
Related #43 (fixed on the Desktop side),
What changed
--gpuis turned intoCUDA_VISIBLE_DEVICESbefore the workers start. The engine itself is unchanged.CUDA_VISIBLE_DEVICESis already set,--gpucan only pick from that list./v1/statsshows the GPU in use undergpus.ft bench bwwrites one profile per card (benchbw/<gpu-uuid>.json); the oldbenchbw.jsonstill works.Breaking change
ft checkpoint --deviceis removed. Use--gpu.Tested on
8x RTX PRO 6000 Blackwell (96 GB), Xeon Platinum 8559C, Amazon Linux 2023, driver 595.71.05, CUDA 13.0, torch 2.11.0+cu130.
Checkpoint: Qwen3.8-27B BF16 as FTW
ft serve --gpu <uuid>,--gpu 6, andCUDA_VISIBLE_DEVICES=2,3 ... --gpu 1: the process lands on the expected card innvidia-smi,/v1/stats.gpusmatches, chat completions work.GPU-deadbeef,99,1,2,"", mixed UUID+index) is rejected with a clear error before loading.ft bench bw --device 6andft daemon start ... -- --gpu <uuid prefix>: the per-card profile is written and/bench/profilereturns the right card.pytest tests/server tests/daemon tests/moe/test_hybrid_fetch.py: 570 passed.Not Tested on Windows or on a box with different GPU models.