Support the LingBot-Video MoE refiner (two-stage high-resolution refinement) - #1546
Open
NancyFyong wants to merge 41 commits into
Open
Support the LingBot-Video MoE refiner (two-stage high-resolution refinement)#1546NancyFyong wants to merge 41 commits into
NancyFyong wants to merge 41 commits into
Conversation
Model code: - LingBotVideoDiT (diffsynth/models/lingbot_video_dit.py) video denoiser - Qwen3-VL text-encoder wrapper (lingbot_video_text_encoder.py) - T2V/V2V pipeline (diffsynth/pipelines/lingbot_video.py) - LingBotVideoUniPCScheduler (FlowMatchScheduler subclass): UniPC multistep for inference, flow-matching schedule for training - Reuse QwenImageVAE via an additive, backward-compatible 5D-video encode/decode path in qwen_image_vae.py - Register models + state-dict converters in model_configs.py Training (issue's core deliverable): - LingBotVideoTrainingModule (examples/lingbot_video/model_training/train.py) with the flow-matching SFT objective - Attention-only LoRA launch script (to_q,to_k,to_v,to_out; MoE/FFN/router left frozen) Examples + docs: - T2V/V2V inference and low-VRAM examples - examples/lingbot_video/README.md Part of the SFT training support for modelscope#1530. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… LingBot-Video
- docs/{en,zh}/Model_Details/LingBot-Video.md + index.rst toctree entries
- two-stage prompt rewriter (structured JSON captions) + offline caption rewrite tool
- LoRA training validation script
- relocate low-VRAM inference into model_inference_low_vram/
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Register the MoE 30B-A3B DiT (128 experts, top-8 group-limited routing, 1 shared expert) and add VRAM management module maps for the LingBot-Video DiT and text encoder, which previously had no entry and fell back to whole-model wrapping. The MoE experts store their weights as grouped nn.Parameter rather than nn.Linear, so the expert container itself is wrapped -- without that the experts, which are the bulk of the model, would stay resident. Also fixes three issues surfaced while validating the MoE path: - _run_grouped_experts guarded the fast path with hasattr(torch, "_grouped_mm"), which is True on CPU even though the kernel is CUDA-only. Now device-aware. - The block and DiT forwards derived the activation dtype from .weight.dtype. Under VRAM management the wrapped layer holds its weight in the offload dtype (or on meta for disk offload), so this read the wrong dtype. Resolved from computation_dtype instead. - The fp32-pinned timestep and modulation MLPs were fed fp32 activations while VRAM management wraps every nn.Linear at computation_dtype, raising a dtype mismatch on any low-VRAM run. Resolved per-layer. The dense low-VRAM example set only vram_limit, which is a no-op without offload_dtype/offload_device, so VRAM management never activated; corrected along with the same claim in the docs. Validated against the official implementation with identical weights: router selection and gate scores are bit-exact, and the full DiT matches bit-exactly at fp32 (bf16 differs only because diffusers' .to() downcasts the fp32-pinned modules). The registered model_hash matches the released checkpoint headers across all 977 keys with no shape mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t rewriter - Scheduler: use FlowMatchScheduler (Wan template) instead of a bespoke UniPC scheduler; flow_match.py restored byte-identical to main. - VAE: restore QwenImageVAE to upstream; the 5D-video encode/decode and latent normalisation now live in LingBotVideoPipeline (encode_video / decode_video). - Rewriter: keep only normalize_caption in diffsynth core; move the two-stage prompt rewriter + system prompts into the inference examples. - Examples: merge into one lingbot-video-dense-1.3b.py (T2V / V2V / optional rewrite); drop the separate _rewrite.py. - Training .sh: use --model_id_with_origin_paths instead of local --model_paths. - Docs: sync en/zh Model_Details + example README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Under low-VRAM offload the AutoWrapped modules compute weights in bf16 but do not cast their inputs, and reading `.weight.dtype` from outside the wrapper returns the resident fp8 dtype. The parameter-free sinusoidal `time_proj` always returns fp32, so feeding it straight into the bf16 `time_embedder` MLP raised "mat1 and mat2 must have the same dtype, but got Float and BFloat16"; `proj_out` had the mirror bug via `self.proj_out.weight.dtype` (fp8 under offload). Cast both inputs to the running compute dtype (`joint`) instead, which is bf16 under offload and matches the weight dtype in a full-precision run. No effect on the normal (non-offload) path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The trimmed diffsynth/pipelines/lingbot_video_prompt_rewriter.py held only
normalize_caption (caption -> compact-JSON serialisation) after the prompt
rewriter was moved to the examples. Its sole core consumer is the pipeline,
which already calls it internally, so keeping a separate one-function module
added a misleadingly-named file ("prompt_rewriter" that no longer rewrites)
for no benefit.
Move normalize_caption (and its _serialize_caption/_caption_from_sample
helpers) into diffsynth/pipelines/lingbot_video.py as module-level functions
and delete the old module. All importers now pull it from lingbot_video; the
example inference/training scripts already imported the pipeline, so this only
tidies their imports. The prompt-rewriting logic stays out of core, in
examples/lingbot_video/model_inference/prompt_rewriter.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Companion to the previous commit, which only recorded the deletion of lingbot_video_prompt_rewriter.py. This commit adds normalize_caption (and its _serialize_caption/_caption_from_sample helpers) into diffsynth/pipelines/lingbot_video.py as module-level functions, and repoints every importer (the pipeline itself, the model_inference / low_vram example scripts, the example prompt_rewriter, train.py and rewrite_captions.py) at diffsynth.pipelines.lingbot_video. Restores a working tree: the two commits together move the single remaining function into the pipeline module. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collapse the AI-generated multi-paragraph banners and docstrings to concise one-line "why" comments matching DiffSynth's light comment style. No code changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ixes to the MoE branch
Brings the reviewer-mi804 integration fixes from lingbot_sft onto the MoE-30B-A3B
branch so both variants share one conventions-aligned implementation:
- Scheduler: drop LingBotVideoUniPCScheduler, use FlowMatchScheduler (Wan template);
diffsynth/diffusion/flow_match.py restored byte-identical to main.
- VAE: restore diffsynth/models/qwen_image_vae.py byte-identical to main; the 5D-video
encode/decode lives in the pipeline (encode_video / decode_video).
- Rewriter: move the prompt-rewriting engine out of diffsynth/ into
examples/lingbot_video/model_inference/{prompt_rewriter,system_prompts}.py; the core
keeps only normalize_caption, folded into the pipeline module.
- Examples/docs/training .sh updated to model_id_with_origin_paths and the merged layout.
Conflict resolutions:
- lingbot_video_dit.py: kept the MoE resolve_bulk_dtype() dtype handling (superset of
the sft joint.dtype fix, and consistent with the rest of the file), plus sft's trimmed
comments and the router fp32 pin.
- low-VRAM dense example + docs: kept sft's corrected vram_limit wording, kept the MoE
section and MoE-specific files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
time_embedder is fp32-pinned (LINGBOT_VIDEO_FP32_MODULES), so under a standard load its weights stay fp32 while joint (the bulk hidden state) is bf16. Casting the fp32 timestep_proj to joint.dtype fed bf16 into the fp32 Linear, raising "mat1 and mat2 must have the same dtype" on any non-offload run. Cast to the layer's own weight dtype instead: fp32 under standard load, bf16 under low-VRAM offload (where the wrapper casts these MLPs to the compute dtype). proj_out keeps joint.dtype since it is a bulk layer held in fp8 under offload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kept the MoE-branch resolve_bulk_dtype() helper for the time_embedder input cast (handles VRAM-wrapped Linears); the sft-branch used a plain .weight.dtype which the MoE version supersedes.
Condition generation on a first frame via a new `input_image` argument. Dense-1.3B reuses its T2V checkpoint (DiT unchanged, in_channels=16); the condition frame is used twice, matching the original LingBot-Video i2v pipeline: fed to the Qwen3-VL text encoder as a visual reference (image tokens prepended to the prompt), and VAE-encoded to a clean latent pinned into the first temporal slot before sampling and after every scheduler step. - LingBotVideoUnit_ImageEmbedder builds the cond latent (via encode_video) and the smart-resized VLM image; no-op for T2V/V2V. - LingBotVideoUnit_PromptEmbedder passes the image to the processor when present. - Ported smart_resize / cover-resize+center-crop / _vlm_image helpers. - Example lingbot-video-dense-1.3b_ti2v.py + released first frame and caption. - EN/ZH docs and example README document the TI2V path and `input_image`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
t2i is text-to-video with num_frames=1 through the same pipeline and DiT (no separate image weight), matching the official runner which sets num_frames=1 and swaps in the still-image negative prompt. - Add DEFAULT_NEGATIVE_PROMPT_IMAGE (verbatim from the official pipeline): drops the temporal/motion terms that cannot apply to a single frame. - Example lingbot-video-dense-1.3b_t2i.py + released still-image caption prompts/t2i_example.json; saves the single returned frame as PNG. - EN/ZH docs and example README document the t2i path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- model_inference_low_vram: add ti2v (input_image) and t2i (num_frames=1 + DEFAULT_NEGATIVE_PROMPT_IMAGE) low-VRAM variants, mirroring the t2v script. - train.py: add opt-in --first_frame_as_condition for image-to-video LoRA. It conditions each clip on its own first frame; FlowMatchSFTLoss already pins the clean first-frame latent and excludes it from the loss, so no core change is needed. A distinct condition column still works via --extra_inputs input_image. - model_training: add ti2v LoRA launch script and validate_lora example. - docs (en/zh) + example README: reference the new scripts and TI2V LoRA. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… reviewer's refactor (531ba56) - TI2V + T2I inference / low-VRAM examples rewritten to the new pipeline API: read structured captions from JSON, pass pipe.default_negative_prompt(_image), drop the removed module-level normalize_caption / DEFAULT_NEGATIVE_PROMPT_IMAGE imports. - Add default_negative_prompt_image attribute to LingBotVideoPipeline (T2I variant with temporal terms removed) so t2i examples can reference it symmetrically with default_negative_prompt. - TI2V LoRA training script aligned to the reviewer's new t2v LoRA (dataset path, num_frames=81, num_epochs=5, dataset_repeat=50); validate_lora rewritten to use pipe.default_negative_prompt + json.load. - New TI2V full-parameter training script + validate_full script (parallel to the reviewer's t2v full training), toggled via --first_frame_as_condition. - Docs rewritten to match the standard Model_Details template: single-checkpoint overview covering T2V / TI2V / T2I with 3-row Examples table, input_image param documented, prompt-rewriter moved under Model Inference, VAE-internals text dropped per reviewer's "keep it about model info + usage" directive. - README.md / README_zh.md: add Update History entry and Video Synthesis series block (Quick Start + Examples table) for LingBot-Video.
Low-VRAM TI2V feeds the condition frame through the Qwen3-VL vision tower, but the LingBotVideoTextEncoder VRAM-management map did not cover its module types. The vision LayerNorm / patch-embed weights stayed on `meta` and the vision RoPE `inv_freq` (a non-persistent buffer, absent from the checkpoint) stayed on CPU, so a TI2V run under offload died with a cuda/cpu device mismatch. T2V / T2I never touch that path. Add LayerNorm, Qwen3VLVisionPatchEmbed and Qwen3VLVisionRotaryEmbedding to the map. PatchEmbed is wrapped whole (not its inner Conv3d) because its forward reads `self.proj.weight.dtype` to cast the input, which under offload would otherwise pick up the fp8 dtype and crash on the bf16 bias. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The input_image param (the TI2V condition frame) was grouped under a '# Video-to-video' comment. Relabel it '# Image-to-video (TI2V)' to match wan_video.py's param grouping and the file's own TI2V terminology. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Address reviewer comments on lingbot_video.py (:20, :185, :214): - Move the module-level TI2V constants (IMAGE_MIN/MAX_TOKEN_NUM, MAX_RATIO, SPATIAL_MERGE_SIZE) and helpers (smart_resize, _round/_ceil/_floor_by_factor, _pixel_tensor_to_pil) plus the pipeline methods preprocess_cond_image, _vision_patch_size and _vlm_image into LingBotVideoUnit_ImageEmbedder, the only consumer. Move IMG_PROMPT_TEMPLATE into LingBotVideoUnit_PromptEmbedder. - Move the pre-loop first-frame latent pin out of __call__ and into LingBotVideoUnit_ImageEmbedder.process; reorder self.units so InputVideoEmbedder (which produces `latents`) runs before ImageEmbedder (which pins into it) and before PromptEmbedder (which consumes vlm_image). __call__ keeps only the per-step re-pin inside the denoise loop. Behavior-preserving for inference and training (the loss overwrites `latents` and does its own first-frame pin, so the unit-level pin is inert there). Verified on GPU: TI2V pins frame 0 to the condition frame; T2V no-op path OK. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g, keep MoE-30B Brings the lingbot_sft refactor and features onto the MoE branch: - Pipeline: adopt sft's unit-based LingBotVideoPipeline (model-agnostic; the MoE variant runs through it unchanged via self.dit). - New Dense-1.3B examples/scripts: TI2V and T2I inference (+ low-VRAM), full and LoRA training, validate_full/validate_lora; scripts/ relocation. Conflict resolutions: - lingbot_video_dit.py: keep MoE's resolve_bulk_dtype / fp32-pinned modules (incl. router). It is the later superset of sft's 7caae83 x.dtype fix, and the file is built on it throughout. - vram_management_module_maps.py: union both maps. The DiT map keeps MoE's fine-grained entries (GroupedExperts wrapped, NOT the whole block, so the walker doesn't keep all experts resident); the text-encoder map takes sft's Qwen3-VL vision-tower entries for TI2V low-VRAM. - docs (en/zh): integrate the MoE-30B-A3B sections with sft's TI2V/T2I/training content into one coherent doc (4-row model table, T2I-aware params). Style: align the MoE inference examples to the refactored dense layout (section banners, explicit negative_prompt=pipe.default_negative_prompt). Verified in the moe worktree with local dense-1.3b weights: the DiT parses and imports; dense-1.3b normal and low-VRAM (fp8 offload) inference both produce correct frame counts with no dtype/device errors (the 7caae83 regression case); a tiny random MoE DiT forward runs grouped_mm on CUDA with finite output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ventions - lingbot_video_dit.py: drop unused LINGBOT_VIDEO_FP32_MODULES / should_keep_in_fp32, remove docstring and inline comments, keep norm_out_modulation call on one line - model_configs.py: keep only the ModelConfig example comment, collapse extra_kwargs onto a single line - vram_management_module_maps.py: drop explanatory comments - MoE inference examples: remove all comments and section dividers - docs: restore the standard template layout, fold the MoE description into the model overview section - README / README_zh: add the missing MoE row so the examples table matches the docs model overview table
- use the released structured-JSON caption from prompts/t2v_example_1.json instead of free-form prose, matching the Dense T2V / TI2V / T2I examples - switch the low-VRAM example to the Dense disk-offload profile (disk -> cpu fp8 -> cuda bf16, vram_limit - 0.5) - add the video-to-video block to the low-VRAM example so it covers the same tasks as the Dense low-VRAM T2V example
…example dataset - T2V examples now download the released structured caption through dataset_snapshot_download, reusing the lingbot-video-dense-1.3b dataset directory, exactly like the Dense T2V examples - add TI2V and T2I MoE examples (inference + low VRAM) mirroring the Dense templates: shared prompts/ captions, assets/ti2v_first_frame.png, default_negative_prompt_image and num_frames=1 for T2I - docs and README: split the MoE row into T2V / TI2V / T2I rows and mention the MoE variant in the release note
Disk offload only materialises weights for module types listed in the VRAM map, so the per-type MoE map left two owners of raw parameters on `meta`: `LingBotVideoBlock.scale_shift_table` and the router weight / correction bias. Both bf16 and low VRAM Dense and MoE inference crashed with "Tensor on device meta is not on the expected device cuda:0". - map `LingBotVideoBlock` and `LingBotVideoRouter` to `AutoWrappedNonRecurseModule`, which manages a module's own parameters while its children keep their individual wrappers, and drop the dead `torch.nn.LayerNorm` entry (`norm_out` has no affine parameters) - cast `scale_shift_table` to the modulation dtype/device in the block forward, following `wan_video_dit.DiTBlock` - keep `e_score_correction_bias` as a non-trainable parameter so the disk loader restores its checkpoint values instead of leaving CPU zeros, and add it in float32 Verified on Robbyant/lingbot-video-moe-30b-a3b: all six MoE examples and the Dense low VRAM examples run, low VRAM peaks at 29.5 GiB versus 74.5 GiB for bf16, and the low VRAM T2I output matches the bf16 output to 16.9/255.
…the MoE branch Upstream squash-merged the Dense-1.3B integration with review changes, so realign the MoE side with what landed: - Drop lingbot_video_text_encoder and its converter; the pipeline now loads Qwen3-VL through Krea2TextEncoder, and the Qwen3-VL vision-tower entries live in the Krea2TextEncoder VRAM map. - Keep a single LingBotVideoDiT VRAM map entry: the MoE variant needs LingBotVideoBlock / LingBotVideoRouter as AutoWrappedNonRecurseModule and LingBotVideoGroupedExperts as AutoWrappedModule so the 128 experts stay individually streamable under offload. - Rename the MoE examples to the upstream _t2v / _ti2v / _t2i scheme and regenerate them line-by-line from the merged Dense examples: Qwen3-VL text encoder + processor configs, captions and condition frames pulled from data/diffsynth_example_dataset, upstream output naming. - Re-add the MoE registry entry, the three MoE rows in both docs tables and both README tables, the MoE paragraph in Model Overview and the release note.
…rectories
Dense and MoE share one example dataset on ModelScope; only
lingbot_video/lingbot-video-dense-1.3b_{t2v,ti2v,t2i}/ exists, so the MoE
examples read the same captions and condition frame instead of a MoE-specific
directory that was never published.
The bulk dtype is already the dtype of the incoming hidden states, and the wrapped linears cast their own weights, so the helper only restated what x.dtype / joint.dtype already carry. Removing it makes the file identical to the Dense implementation except for the offload fixes; TI2V output stays bit-identical in bf16 and under low-VRAM offload.
…fload
AutoWrappedNonRecurseModule leaves parameter casting to the model, and the
grouped-expert matmuls read experts.w1/w2/w3 directly instead of going through
a forward, so an AutoWrappedModule wrapper never got the chance to cast them.
With a vram_limit that actually forces layers to stay on CPU, the router
projection and the expert matmuls therefore hit CPU weights:
RuntimeError: Expected all tensors to be on the same device,
but got mat2 is on cpu
Move the experts to AutoWrappedNonRecurseModule as well and cast the router
weight, the correction bias and the three expert tensors to the token device
and dtype at use, matching how scale_shift_table is already handled. bf16 and
default low-VRAM outputs stay bit-identical.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Collaborator
|
@NancyFyong The |
Contributor
Author
I‘ll fix it quickly, thank you for review! |
# Conflicts: # README.md # README_zh.md # diffsynth/configs/model_configs.py # diffsynth/diffusion/flow_match.py # docs/en/Model_Details/LingBot-Video.md # docs/zh/Model_Details/LingBot-Video.md # examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py # examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py # examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py # examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py # examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py # examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py # examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh # examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh # examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh # examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh # examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py # examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py # examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py # examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py
…amples Matches the repo-wide no-decorative-separator rule already applied to the other lingbot_video examples.
NancyFyong
marked this pull request as ready for review
August 10, 2026 09:00
Contributor
Author
Hi,@mi804, this pr is ready, please take a look if you have time. BTW, the bf16 inference model_inference 1088×1920 H20-95 GB is OOM, so it is recommended to use low varm inference. |
mi804
reviewed
Aug 10, 2026
| time_division_factor=4, time_division_remainder=1, | ||
| ) | ||
| self.scheduler = FlowMatchScheduler(template="Wan") | ||
| self.scheduler = FlowMatchScheduler(template="LingBot-Video") |
Collaborator
There was a problem hiding this comment.
The changes to the scheduler also affect the existing Dense and MoE models. Please make sure that both models can still generate videos correctly after these changes.
mi804
reviewed
Aug 10, 2026
| **models, timestep=timestep, progress_id=progress_id | ||
| ) | ||
| inputs_shared["latents"] = self.step(self.scheduler, progress_id=progress_id, noise_pred=noise_pred, **inputs_shared) | ||
| if t_thresh is not None and inputs_shared.get("first_frame_latents") is not None: |
Collaborator
There was a problem hiding this comment.
If the results are similar, this part of the logic could be moved into model_fn to keep the main denoising loop in the __call__ function cleaner. You can refer to this implementation for reference:
mi804
reviewed
Aug 10, 2026
| # Scheduler | ||
| num_inference_steps: int = 40, | ||
| sigma_shift: float = 3.0, | ||
| # Refinement pass: start from a partially noised input video at sigma=t_thresh and |
Collaborator
There was a problem hiding this comment.
The comments in the __call__ function should be kept concise, for example: just High-Resolution Refinement.
mi804
reviewed
Aug 10, 2026
|
|
||
| ## Model Overview | ||
|
|
||
| MoE-30B-A3B is the larger variant: 30B total parameters with ~3B active per token, where each MoE layer holds 128 routed experts plus 1 shared expert and routes every token to 8 experts with group-limited top-k (4 groups, top-2 groups). It serves the same three tasks through the same pipeline, only the model ID and the shard glob change. |
Collaborator
There was a problem hiding this comment.
The model overview table should not include additional explanatory notes. If necessary, a brief description can be added to the example script instead. In most cases, users can find the relevant technical details through the model link, so there is no need to over-explain the technical approach in the documentation.
mi804
reviewed
Aug 10, 2026
|
|
||
| Training captions should be **structured-JSON captions** (the same in-distribution format used at inference). If your dataset stores raw prose, rewrite it once offline with [`examples/lingbot_video/model_training/scripts/rewrite_captions.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/scripts/rewrite_captions.py) before training. | ||
|
|
||
| The MoE-30B-A3B scripts launch with [`examples/lingbot_video/model_training/full/accelerate_config_moe.yaml`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/accelerate_config_moe.yaml) (DeepSpeed ZeRO-2, optimizer CPU offload, bf16), and the full-parameter scripts additionally enable `--use_gradient_checkpointing_offload`. Note that `--lora_target_modules "to_q,to_k,to_v,to_out"` only reaches the attention projections: the routed experts and the router are stored as bare parameter tensors for grouped matmul, so LoRA leaves them frozen. Use full-parameter training to update the expert stack. |
Collaborator
There was a problem hiding this comment.
This part can be removed. Users can adjust the training configuration based on their available VRAM.
mi804
reviewed
Aug 10, 2026
|
|
||
| ## 模型总览 | ||
|
|
||
| MoE-30B-A3B 是更大的版本：总参数量 30B，每个 token 激活约 3B，每个 MoE 层包含 128 个路由专家和 1 个共享专家，并使用 group-limited top-k（4 组，取 top-2 组）将每个 token 路由到 8 个专家。它通过同一条 Pipeline 支持同样的三种任务，只需更换模型 ID 与分片通配符。 |
mi804
reviewed
Aug 10, 2026
|
|
||
| 训练时 `prompt` 字段应存放**结构化 JSON caption**（与推理时使用的分布内格式一致）。如果数据集里存的是原始散文，可先使用 [`examples/lingbot_video/model_training/scripts/rewrite_captions.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/scripts/rewrite_captions.py) 离线改写一次。 | ||
|
|
||
| MoE-30B-A3B 的训练脚本通过 [`examples/lingbot_video/model_training/full/accelerate_config_moe.yaml`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/accelerate_config_moe.yaml) 启动（DeepSpeed ZeRO-2、优化器状态 CPU offload、bf16），全量训练脚本还额外开启了 `--use_gradient_checkpointing_offload`。需要注意的是，`--lora_target_modules "to_q,to_k,to_v,to_out"` 只覆盖注意力投影层：路由专家与路由器以裸参数张量的形式存放以便做分组矩阵乘法，LoRA 不会训练它们。如需更新专家部分，请使用全量训练。 |
mi804
reviewed
Aug 10, 2026
| with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-dense-1.3b_t2v/t2v_example_1.json", "r", encoding="utf-8") as f: | ||
| caption = json.load(f) | ||
|
|
||
| pipe = LingBotVideoPipeline.from_pretrained( |
Collaborator
There was a problem hiding this comment.
It's better removing this part and directly using the existing videos from the dataset.
| caption = json.load(f) | ||
| input_image = Image.open(os.path.join(base, "ti2v_first_frame.png")).convert("RGB") | ||
|
|
||
| pipe = LingBotVideoPipeline.from_pretrained( |
Collaborator
There was a problem hiding this comment.
It's better removing this part and directly using the existing videos from the dataset.
- pipelines/lingbot_video.py: * shorten the refiner param comment to 'High-Resolution Refinement' (mi804 comment modelscope#3) * drop the in-loop step-after re-pin; the refiner reuses the existing 'first_frame_latents' pin inside model_fn_lingbot_video (mi804 comment modelscope#2) - docs/{en,zh}/Model_Details/LingBot-Video.md: remove the explanatory notes next to the model-overview table and inside the training section (mi804 comments modelscope#4/modelscope#6 and modelscope#5/modelscope#7) - examples/lingbot_video/model_inference{,_low_vram}/*_refiner.py: drop the stage-1 base pass and load the reference clip straight from the example dataset (mi804 comments modelscope#8/modelscope#9)
mi804
reviewed
Aug 10, 2026
Collaborator
There was a problem hiding this comment.
If model inference runs into OOM issues, I would suggest enabling CPU offloading in the standard inference script as well. You can use the following vram_config:
vram_config = {
"offload_dtype": torch.bfloat16,
"offload_device": "cpu",
"onload_dtype": torch.bfloat16,
"onload_device": "cpu",
"preparing_dtype": torch.bfloat16,
"preparing_device": "cuda",
"computation_dtype": torch.bfloat16,
"computation_device": "cuda",
}
Collaborator
There was a problem hiding this comment.
If model inference runs into OOM issues, I would suggest enabling CPU offloading in the standard inference script as well. You can use the following vram_config:
vram_config = {
"offload_dtype": torch.bfloat16,
"offload_device": "cpu",
"onload_dtype": torch.bfloat16,
"onload_device": "cpu",
"preparing_dtype": torch.bfloat16,
"preparing_device": "cuda",
"computation_dtype": torch.bfloat16,
"computation_device": "cuda",
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LingBot-Video MoE Refiner (two-stage high-resolution refinement)
Summary
Support the refiner shipped with
Robbyant/lingbot-video-moe-30b-a3b, stacked on the MoE integration in #1545. The refiner is a second 30B-A3B DiT underrefiner/that performs a short second pass at a higher resolution over an already generated clip — the official setup generates at 480x832 with 40 steps, then refines at 1088x1920 with 8 steps.refiner/config.jsonmatchestransformer/config.json, and the two directories have the same key+shape hash (426db1dddf14e8e0f7522d7360745d27, 977 keys / 13 shards), so no new registry entry is needed — the existing MoE entry loads the refiner when the shard glob points atrefiner/. The weights themselves are a fine-tune, not a copy (cosine 0.998 on the attention projections, 0.083 one_score_correction_bias).How it works
The refinement pass is SDEdit at a higher resolution, so it reuses the existing
input_videopath: the base clip is read back at the target resolution, VAE-encoded, and noised tosigma=t_thresh. Two additions make the schedule match the official implementation:FlowMatchSchedulergains a"LingBot-Video"template. Witht_thresh=Noneit is value-identical to theWantemplate it replaces; witht_threshset it truncates the shifted sigma grid at the threshold, pins the first sigma exactly att_thresh, and appendssigma_tail_stepsextra low-noise steps ending atsigma_min.LingBotVideoPipeline.__call__gainst_thresh(defaultNone, i.e. plain generation) andsigma_tail_steps(default2). Whent_threshis set, TI2V re-pins the clean first-frame latent after every scheduler step and conditions on text only, matching the official refiner path.For
num_inference_steps=8, sigma_shift=3.0, t_thresh=0.85, sigma_tail_steps=2this reproduces the officialcompute_refiner_sigmasoutput to 2.4e-08:Usage is a second pipeline load with a different shard glob:
Examples
All paths are relative to
examples/lingbot_video/. Each script runs both stages end to end (base generation, then reload with the refiner shards):model_inference/lingbot-video-moe-30b-a3b_t2v_refiner.pymodel_inference_low_vram/lingbot-video-moe-30b-a3b_t2v_refiner.pymodel_inference/lingbot-video-moe-30b-a3b_ti2v_refiner.pymodel_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v_refiner.pyPrompts and the TI2V condition frame come from the same published example dataset directories the other LingBot-Video examples use. Docs:
docs/en/Model_Details/LingBot-Video.md,docs/zh/Model_Details/LingBot-Video.md(new "Two-stage refinement" section,t_thresh/sigma_tail_stepsparameter entries); both README example tables gain two rows.Results
T2V — 480x832 base (top) vs 1088x1920 refined (bottom), frames 0/20/40/60/80
T2V — detail crop, top bicubic-upscaled base, bottom refiner output
TI2V — 480x832 base (top) vs 1088x1920 refined (bottom), frames 0/20/40/60/80 — the condition frame is re-pinned after every step, so the first refined frame is bit-identical (VAE round trip only, MAE 2.14 / 255) with the input image.
TI2V — detail crop, top bicubic-upscaled base, bottom refiner output
Full-resolution 81-frame clips (open on GitHub for the built-in player): T2V base 480x832 · T2V refined 1088x1920 · TI2V refined 1088x1920
Measured on 1 GPU with the low-VRAM configuration: the 1088x1920x81 pass runs at ~218 s/it for 8 steps and peaks at 53.5 GiB. Laplacian variance at matched resolution is 2.74x (T2V) and 2.50x (TI2V) that of a bicubic upscale of the base clip, while the layout is preserved (MAE 6.45 / 255 between the downscaled refined clip and the base clip). For TI2V, the first output frame differs from the condition image by MAE 2.14 / 255 — the VAE round trip — versus 24.09 for the base path, which pins the condition latent only once.
Dependencies
No new packages, no new model registry entries.
Notes
t_thresh=Nonereturns exactly the same sigmas and timesteps as theWantemplate for the configurations the pipeline and trainer use. End to end, the low-VRAM MoE T2V, V2V and TI2V examples produce byte-identical outputs before and after this change (md5d3017c72…,ab05029b…,67896646…respectively; T2V rerun twice to confirm the comparison is meaningful).--reuse_condition_features,--batch_cfg, context parallelism, and an FP8 fused-MoE runtime. Those are orthogonal to this PR.