[FEATURE] Run Wan2.2 pipelines on Ascend NPU - #42
Open
yx0716 wants to merge 7 commits into
Open
Conversation
The VAE weight-side conversion to channels_last_3d is already guarded by cuDNN availability, but the activation-side casts in CausalConv3d.forward and the spatial-parallel halo conv were unconditional. NPU and CPU reject channels_last_3d activations, so Wan2.2 VAE forward failed on Ascend with ERR01007 ("NPU contiguous operator only supported contiguous memory format"). Route non-CUDA tensors through standard contiguous instead.
Verified: Wan2.2-TI2V-5B text-to-video smoke on Ascend 910B2 (CANN 8.2, torch_npu 2.9.0) completes end to end; CUDA path unchanged.
create_device_mesh_from_config defaulted device_type to "cuda" and 12 of 14 call sites rely on the default, so parallel denoising on NPU crashed inside torch DeviceMesh with "integer modulo by zero" (zero visible CUDA devices). Resolve the default from current_platform; explicit arguments keep working. Verified: 4-card (cfg=2 x ulysses=2) Wan2.2-TI2V-5B run on Ascend 910B2 builds the [cfg, ulysses] mesh over hccl; on CUDA the default resolves to "cuda" as before.
Parallel workers exchange request and result tensors over torch.multiprocessing queues, which relies on device IPC. torch_npu cross-process sharing raised "devptr INTERNAL ASSERT FAILED ... entry in cache has missing shared_ptr" when rebuilding queued NPU tensors. Enable the existing queue_with_cpu marshalling by default on non-CUDA platforms, mirror it on the result path, and move results back to the stage device in the main process. Worker dispatch already moves inputs to the local device, so the contract is unchanged and CUDA keeps direct device-queue transport. Verified: 4-card Wan2.2-TI2V-5B denoising on Ascend 910B2 completes with workers exiting cleanly.
ops/fp8_attention.py defined @triton.jit kernels at module scope, making "import triton" unconditional for every consumer of the wan_video and minimax DiT import chains even though the fused path is CUDA-only by contract. Move the two kernels and their launcher to telefuser/kernel/triton/fp8_attention.py and import them lazily inside quantize_fp8_qkv after its existing CUDA validation, matching the established ops -> kernel.triton dispatch pattern (see ops/rotary.py and kernel/__init__.py). The pure-torch quantize/dequantize helpers are unchanged. Verified: wan22 pipeline import and Wan2.2-TI2V-5B smoke succeed on an NPU host without triton installed; tests/unit/models/test_wan_video_sol_attention.py passes with CUDA-only cases skipped.
recv, recv_latent, and recv_latent_async allocated receive buffers with device="cuda", which fails on NPU-only hosts before irecv can run. Allocate on current_platform.device_type (identical behavior on CUDA) and parameterize the test tensor device the same way so the suite runs on CUDA, NPU, and CPU hosts. Verified: tests/unit/distributed/test_pp_comm.py 12 passed on Ascend 910B2 (previously 6 device-related failures).
The non-CUDA result path introduced for CPU-marshalled queues moved results back to self._stage.device inside _wait_result. That breaks worker unit tests on non-CUDA hosts (mocked stages resolve .device through the method proxy or to a MagicMock), and it is unnecessary: stage entry points already place their inputs, and the Wan VAE decode paths move latents themselves. Drop the move so _wait_result returns results untouched; workers still marshal results through CPU when device IPC is unavailable. Verified: tests/unit/worker passes on an Ascend host (same non-CUDA branch as CPU CI) and the 4-card Wan2.2-TI2V-5B smoke still completes; ruff clean.
The example hardcoded device="cuda", so running it on an NPU host required editing the script. Default the pipeline device to current_platform.device_type so the same command runs unmodified on CUDA and NPU; on CUDA hosts this resolves to "cuda" as before. Verified: python examples/wan_video/wan22_t2v_5b.py --gpu_num 1 and --gpu_num 4 run unmodified on Ascend 910B (50 steps, 121 frames, default prompt).
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enable the Wan2.2 pipelines to run on Ascend NPU, single-card and multi-card, with a minimal correctness-focused change set. All fixes reuse the existing platform architecture (
current_platform, capability gates,queue_with_cpu); CUDA behavior is unchanged by construction — every new branch is gated on platform or tensor device type.Motivation
TeleFuser already ships a platform abstraction (
telefuser/platforms/,CustomOp.forward_npu,dist_backend="hccl"), and the wan22 pipeline code itself is platform-clean. In practice, a handful of CUDA defaults outside that abstraction still prevented NPU execution: an unconditionalchannels_last_3dactivation cast, a"cuda"device-mesh default, device-IPC-dependent worker queues, a hardimport triton, and"cuda"-allocated P2P buffers. This PR removes those blockers.Type of Change
Changes Made
models/wan_video_vae.py,distributed/vae_spatial.py: gate activation-sidechannels_last_3dcasts to CUDA tensors (weight-side conversion was already cuDNN-gated; NPU rejects that memory format withERR01007).distributed/device_mesh.py:create_device_mesh_from_configdefaultsdevice_typetocurrent_platform.device_type(12 of 14 call sites rely on the default; the"cuda"default crashed torchDeviceMeshon NPU hosts withinteger modulo by zero). Explicit arguments keep working.worker/parallel_worker.py: enable the existingqueue_with_cpumarshalling by default on non-CUDA platforms and mirror it on the result path (torch_npu cross-process tensor sharing raisesdevptr INTERNAL ASSERT FAILEDwhen rebuilding queued NPU tensors). Worker dispatch already moves inputs to the local device, so the contract is unchanged; CUDA keeps direct device-queue transport.ops/fp8_attention.py→kernel/triton/fp8_attention.py: move the@triton.jitfused FP8 QKV kernels intokernel/triton/and import them lazily insidequantize_fp8_qkvafter its existing CUDA-only validation, matching the establishedops -> kernel.tritondispatch pattern. The wan_video / minimax DiT import chains no longer require triton to be installed.distributed/pp_comm.py,tests/unit/distributed/test_pp_comm.py: allocate P2P receive buffers oncurrent_platform.device_typeinstead of hard-coded"cuda", and parameterize the test tensor device so the suite runs on CUDA, NPU, and CPU hosts.examples/wan_video/wan22_t2v_5b.py: default the pipeline device to the detected platform so the example runs unmodified on CUDA and NPU hosts (resolves to"cuda"on CUDA as before).Testing
pytest tests/)Test commands:
Results on Ascend 910B: single-card Wan2.2-TI2V-5B text-to-video succeeds end to end (bf16,
TORCH_SDPA, unipc; semantically correct output); theMODEL_CPU_OFFLOADvariant is bit-identical to the no-offload run (same seed). 4-card denoising completes with workers exiting cleanly, using the existing eager ulysses fallbacks.tests/unit/distributed/test_pp_comm.pypasses 12/12 (previously 6 device-related failures on non-CUDA hosts), andtests/unit/models/test_wan_video_sol_attention.pypasses with CUDA-only cases skipped. CUDA hosts execute identical code paths as before (device-mesh default resolves to"cuda",queue_with_cpustays opt-in, activation casts and buffer devices are unchanged on CUDA).Checklist
ruff)pre-commit run --all-files— ruff + ruff-format; also covered by the lint CI job)pytest tests/)test_pp_comm.py)[TYPE] Brief descriptionRelated Issues
N/A
Additional Notes
Performance parity is intentionally out of scope: with everything on eager fallbacks, multi-card currently shows limited speedup over single-card on NPU (per-layer eager all_to_all over hccl and the serial VAE decode dominate). A follow-up PR will address NPU-optimized paths behind the existing dispatch points.
offload/async_offload.pyremains CUDA-only (auto-disabled elsewhere); Wan2.2 A14B (MoE dual-DiT) shares these code paths but has not been exercised on NPU hardware yet.GPU Architecture Support
No CUDA kernels are added or modified — the fused FP8 QKV kernels are moved verbatim and their SM90 targeting is unchanged — so no architecture box is checked.
Device support added by this PR: Ascend NPU (verified on Atlas 910B, CANN 8.2, torch 2.9.0 + torch_npu 2.9.0.post1, via the existing
telefuser/platforms/NPU platform). CUDA devices are unaffected.Performance Impact
None on CUDA (all new branches are platform- or device-gated; CUDA executes the previous code paths).
Ascend 910B baseline established by this PR — Atlas 910B (8× 910B2 64 GB), CANN 8.2, torch 2.9.0 + torch_npu 2.9.0.post1. All numbers are produced by the unmodified example
examples/wan_video/wan22_t2v_5b.py(bf16,TORCH_SDPA, unipc, 50 denoising steps,MODEL_CPU_OFFLOAD,cfg_scale=5.0— i.e. two DiT forwards per denoising step; under cfg parallelism the two guidance branches run on separate ranks). Times are end-to-end generate (one-time init excluded); every row is reproducible with the exact command shown:python examples/wan_video/wan22_t2v_5b.py --gpu_num Npython examples/wan_video/wan22_t2v_5b.py --gpu_num N --resolution 720pMeasured stage split at 480p / 121 frames (from run-log timestamps):