Skip to content

[tests] A tester mixin for running modular blocks on their own #15267

Open

Description

@yiyixuxu

A modular pipeline is meant to be used in pieces: you can take the text encoder block, turn it into its own pipeline, and run it separately from the rest. We want that to keep working for every pipeline, and we want it to be intuitive to use that way. Today each test file checks the first by hand, if at all, and nothing checks the second.

What we want to test

For each top-level block of a pipeline (text encoder, VAE encoder, denoise, decode, ...):

  • it builds into its own pipeline with init_pipeline()
  • that pipeline loads only the components the block declares, e.g. just vae for a VAE encoder
  • it runs on the inputs it declares and returns the outputs it declares
  • running the blocks one after another, each fed the outputs of the ones before it, gives the same result as the assembled pipeline
  • it is intuitive to use on its own: what you pass in and what you get back make sense without knowing the rest of the pipeline

Is it intuitive to use this way?

Running image -> encoder -> denoise -> decoder by hand shows interface problems that the assembled pipeline hides. Patterns we have run into so far:

  • Components it does not own. It would be weird for a VAE encoder block to require the transformer just because it reads a value from its config. A block run on its own should load only what it actually runs.
  • Too much crossing a boundary. A decoder that takes packed latents also needs latent_height and latent_width to unpack them, so the user has to carry those over from the denoise step. We fixed this for flux, krea2 and ltx by making unpack part of the denoise group ([modular] move unpack out of the decoder blocks for flux, krea2 and ltx #14730, move unpack out of the flux, krea2 and ltx decoders into the core denoise groups #14737). What one block hands to the next should be a short, canonical list (prompt_embeds, image_latents, latents).
  • Optional in, something out. A text encoder block should not require the guider: the user should not have to configure guidance in two separate places, once for the text encoder and once for the denoiser. Run on its own, it gives back negative embeddings when you pass a negative_prompt and none when you don't. Getting a negative_prompt=None encoded anyway is unintuitive ([modular] guider should be an optional component for text encoder blocks #14469).
  • Redundant inputs. A denoise block that takes image_latents should not also need height and width, unless the pipeline supports an output size different from the reference's.

In this test we want to check for these specific patterns. We catch new ones over time during reviews, so the mixin should make adding one cheap.

What it looks like today

Fifteen test files under tests/modular_pipelines/ build a one-block pipeline by hand, each with its own setup. For example, from the MiniMax H3 Image tests:

class TestMiniMaxH3ImageModularBlocks(MiniMaxH3ImageModularPipelineTesterConfig):
    def test_vae_encoder_block_standalone(self):
        pipe = MiniMaxH3ImageEditVaeEncoderStep().init_pipeline(self.pretrained_model_name_or_path)
        pipe.load_components(dtype=torch.float32)

        reference_latents = pipe(
            reference_images=[Image.new("RGB", (64, 32))],
            generator=self.get_generator(0),
            output="reference_latents",
        )

        assert set(pipe.components) == {"vae"}
        assert len(reference_latents) == 1
        assert reference_latents[0].shape == (1, 4, 8, 16)

The build, the load, the component check and the "does it return its output" check are the same for every block in every pipeline. Only the last assertion is specific to this model.

Proposal

A tester mixin next to the existing ones in tests/modular_pipelines/testing_utils/, composed with the same config class as the others. It walks the top-level blocks of each workflow and runs the checks above, so a new pipeline gets them without writing anything. Model-specific assertions stay as methods on the pipeline's own test class.

How to contribute

This issue carries the idea-contribution-welcome label. We are not currently looking for a PR here, we are looking for your feedback, especially if you use modular blocks on their own:

  • other patterns you think this test should check for
  • places where running a block standalone is unintuitive today: an input you did not expect to need, an output you did not expect to get, a value you had to carry from one block to the next

we may invite you to work with us on this once we agree on the implementation. If we decide to write the PR ourselves, anyone whose feedback shaped it will be added as a co-author.


Posted by Claude on behalf of @yiyixuxu. The text was drafted together and approved by yiyixuxu.

Activity

  1. yiyixuxu commented on Nov 12, 2026

    @yiyixuxu
    CollaboratorAuthor

    cc @sayakpaul let me know what you think

  2. added
    idea-contribution-welcomeWe want discussion, not a PR. feedbacks that help us will get co-author credit
    on Nov 12, 2026
  3. sayakpaul commented on Nov 16, 2026

    @sayakpaul
    Member

    Good idea!

    A tester mixin next to the existing ones in tests/modular_pipelines/testing_utils/, composed with the same config class as the others. It walks the top-level blocks of each workflow and runs the checks above, so a new pipeline gets them without writing anything. Model-specific assertions stay as methods on the pipeline's own test class.

    I think we should allow defining the list of these modular blocks we want to test for each pipeline under src/diffusers/modular_pipelines. We usually define these blocks under a modular_blocks_*.py file, pertaining to a pipeline. For example:

    AUTO_BLOCKS = InsertableDict(
        [
            ("prompt_upsampler", Flux3AutoPromptUpsampleStep()),
            ("text_encoder", Flux3TextEncoderStep()),
            ("vae_encoder", Flux3AutoVaeEncoderStep()),
            ("denoise", Flux3AutoCoreDenoiseStep()),
            ("decode", Flux3DecodeStep()),
        ]
    )

    Do we want to stick to these blocks only? I think these should suffice. Each of these blocks has different inputs, different outputs, and shapes to check against. I think these should be the atomic tests the mixin would have. Then on the point of

    Redundant inputs. A denoise block that takes image_latents should not also need height and width, unless the pipeline supports an output size different from the reference's.

    We can define properties at the mixin level (so that they can be overridden) to run these negations. So, a rough skeleton for this mixin could be like so:

    class ModularBlocksTesterMixin:
        # The only values one block may hand to the next. Anything else a block reads has to come from the user.
        handoff_names = frozenset(["prompt_embeds", "negative_prompt_embeds", "image_latents", "latents"])
        # Components a block must not load when it runs on its own.
        unowned_components = {
            "text_encoder": {"guider", "transformer", "unet", "vae"},
            "vae_encoder": {"guider", "transformer", "unet"},
            "decode": {"guider", "transformer", "unet"},
        }
        # Inputs a block must run without, even when the user passes them to the assembled pipeline.
        redundant_inputs = {"decode": {"height", "width"}}
        # Optional input -> (a value for it, the output that must be None when the input is not passed).
        optional_outputs = {"negative_prompt": ("blurry", "negative_prompt_embeds")}
    
        def get_block_pipeline(self, block_name):
            block = self.pipeline_blocks_class().sub_blocks[block_name]
            pipe = block.init_pipeline(self.pretrained_model_name_or_path)
            pipe.load_components(dtype=torch.float32)
            return pipe.to(torch_device)
    
        def get_inputs(self, pipe, handoffs=None, skip=()):
            """The dummy inputs `pipe` declares, every optional input given, plus what earlier blocks handed
    over."""
            optional = {name: value for name, (value, _) in self.optional_outputs.items()}
            inputs = {**self.get_dummy_inputs(), **optional, **(handoffs or {})}
            return {k: v for k, v in inputs.items() if k in pipe.blocks.input_names and k not in skip}
    
        def test_blocks_load_only_their_components(self):
            for block_name in self.pipeline_blocks_class().sub_blocks:
                pipe = self.get_block_pipeline(block_name)
                unowned = set(pipe.components) & self.unowned_components.get(block_name, set())
                assert not unowned, f"Block '{block_name}' loads components it does not own: {sorted(unowned)}"
    
        def test_blocks_chained_match_pipeline(self):
            pipe = self.pipeline_blocks_class().init_pipeline(self.pretrained_model_name_or_path)
            pipe.load_components(dtype=torch.float32)
            pipe.to(torch_device)
            expected = pipe(**self.get_inputs(pipe), output=self.output_name)
    
            # one generator for the whole chain, as in the assembled pipeline
            handoffs = {"generator": self.get_generator(0)}
            for block_name in self.pipeline_blocks_class().sub_blocks:
                block_pipe = self.get_block_pipeline(block_name)
                outputs = block_pipe(
                    **self.get_inputs(block_pipe, handoffs, skip=self.redundant_inputs.get(block_name, ())),
                    output=block_pipe.blocks.intermediate_output_names,
                )
                handed_over = self.handoff_names | {self.output_name}
                handoffs.update({k: v for k, v in outputs.items() if k in handed_over and v is not None})
    
            assert_tensors_close(handoffs[self.output_name], expected, atol=1e-3, rtol=1e-3)
    
        def test_optional_input_gates_its_output(self):
            for block_name, block in self.pipeline_blocks_class().sub_blocks.items():
                for input_name, (value, output_name) in self.optional_outputs.items():
                    if input_name not in block.input_names:
                        continue
                    pipe = self.get_block_pipeline(block_name)
                    inputs = self.get_inputs(pipe, skip={input_name})
                    assert pipe(**inputs, output=output_name) is None, (
                        f"Block '{block_name}' returns `{output_name}` without `{input_name}`"
                    )
                    assert pipe(**inputs, **{input_name: value}, output=output_name) is not None

    Then each block-level test suite could subclass from the mixin and override as needed.

  4. charchit7 commented on Nov 17, 2026

    @charchit7
    Contributor

    I have been tinkering with qwen-image for a use-case, so I ran the above sketch as written against the merged QwenImage blocks (hf-internal-testing/tiny-qwenimage-modular, CPU).

    It catches the guider / negative-prompt pattern in the text encoder as intended. Two things came up in test_blocks_chained_match_pipeline:

    1. Legitimate hand-offs outside the canonical list. It fails with Required input 'prompt_embeds_mask' is missing. QwenImage's text encoder hands prompt_embeds_mask and negative_prompt_embeds_mask to denoise, because the transformer needs them, and they aren't in handoff_names. The error names the input but not that the mixin filtered it out. Asserting up front that everything a block reads from earlier blocks is in handoff_names, plus a per-pipeline extra_handoff_names, would give a message like "denoise reads prompt_embeds_mask from text_encoder, which is not a declared hand-off", and each exception becomes a reviewable line in the test file.

    2. A pattern that isn't on the list yet: config from a component the block doesn't own, with a silent default. With the masks added to handoff_names, the test still fails: Shape mismatch: actual [1, 3, 16, 16] vs expected [1, 3, 32, 32]. The standalone denoise pipeline has no vae, so vae_scale_factor falls back to 8, while the tiny VAE's factor is 4. The latents come out as (1, 4, 1, 4, 4) where the assembled pipeline gives (1, 4, 1, 8, 8). With the real VAE the factor is 8, so it's invisible outside the tiny repo.

    I understand the fallback was required in #14943 added it for Anima so the denoise block runs without a VAE. The cost is that it's only right when the VAE matches the default. With the real QwenImage VAE the factor is 8, so it's invisible outside the tiny repo, but a finetune or a different VAE with another factor would get the wrong latent size with no error.

    So, for the mixin, this is the flip side of "components it does not own": the block doesn't require the VAE, but it guesses a value from it. The chained test catches it only because the tiny model happens to differ from the default. A direct check could compare the derived properties a block uses (vae_scale_factor, num_channels_latents, ...) between the block pipeline and the assembled one. How to resolve a mismatch is a separate question, e.g. letting the user pass the value, or handing it over with the latents.

    It isn't specific to QwenImage. Running the same two steps on other tiny repos:

    Pipeline Extra hand-off Canonical hand-offs only vae_scale_factor standalone / assembled Latents standalone / assembled
    Krea 2 prompt_embeds_mask, negative_prompt_embeds_mask Required input 'prompt_embeds_mask' is missing 8 / 4 (1, 4, 1, 4, 4) / (1, 4, 1, 8, 8)
    Flux pooled_prompt_embeds AttributeError: 'NoneType' object has no attribute 'repeat' 8 / 1 (1, 1, 0, 0) / (1, 1, 8, 8)
    Z-Image none OK 8 / 2 (1, 16, 4, 4) / (1, 16, 16, 16)

    Repro for both (tiny repo, CPU):

    script
    import torch
    
    from diffusers.modular_pipelines import QwenImageAutoBlocks
    
    
    repo = "hf-internal-testing/tiny-qwenimage-modular"
    blocks = QwenImageAutoBlocks()
    
    
    def build(block):
        pipe = block.init_pipeline(repo)
        pipe.load_components(dtype=torch.float32)
        return pipe
    
    
    full = build(blocks)
    text_encoder = build(blocks.sub_blocks["text_encoder"])
    denoise = build(blocks.sub_blocks["denoise"])
    
    text_outputs = ["prompt_embeds", "negative_prompt_embeds", "prompt_embeds_mask", "negative_p
    embeds = text_encoder(prompt="dance monkey", negative_prompt="bad quality", max_sequence_length=16, output=text_outputs)
    denoise_inputs = {"num_inference_steps": 2, "height": 32, "width": 32}
    
    # 1. Only the canonical hand-offs: the masks are missing.
    canonical = {k: v for k, v in embeds.items() if k in ("prompt_embeds", "negative_prompt_embeds")}
    try:
        denoise(**canonical, **denoise_inputs, generator=torch.manual_seed(0), output="latents")
    except ValueError as e:
        print("canonical hand-offs only:", e)
    
    # 2. With the masks handed over: the standalone block has no VAE and falls back to the default scale factor.
    latents = denoise(**embeds, **denoise_inputs, generator=torch.manual_seed(0), output="latent
    expected = full(
        prompt="dance monkey",
        negative_prompt="bad quality",
        max_sequence_length=16,
        **denoise_inputs,
        generator=torch.manual_seed(0),
        output="latents",
    )
    print("vae_scale_factor  standalone denoise:", denoise.vae_scale_factor, "| assembled:", full.vae_scale_factor)
    print("latents           standalone denoise:", tuple(latents.shape), "| assembled:", tuple(e
    canonical hand-offs only: Required input 'prompt_embeds_mask' is missing
    vae_scale_factor  standalone denoise: 8 | assembled: 4
    latents           standalone denoise: (1, 4, 1, 4, 4) | assembled: (1, 4, 1, 8, 8)
    
  5. sayakpaul commented on Nov 17, 2026

    @sayakpaul
    Member

    Can you simplify the explanation and try to write it in your own words? The extra slop creates a bit of fatigue.

  6. charchit7 commented on Nov 17, 2026

    @charchit7
    Contributor

    Yeah, sorry my bad,

    So, I ran your sketch on QwenImage with the tiny repo and two things came up in the chained test.

    1. The test doesn't pass every output forward, it keeps only the names in handoff_names. In QwenImage the text encoder also gives prompt_embeds_mask and negative_prompt_embeds_mask to denoise. So the test drops the two masks and denoise fails with "Required input 'prompt_embeds_mask' is missing". To get past it I added the two names in the test subclass:
    handoff_names = ModularBlocksTesterMixin.handoff_names | {"prompt_embeds_mask", "negative_prompt_embeds_mask"}

    Is that the intended way for pipelines that need extra handoffs? Please correct me if I am missing something.

    1. After I added the masks, the output was 16x16 instead of 32x32. The denoise block decides the latent size from vae_scale_factor so when it runs alone it has no vae, it uses the default 8. The tiny vae is 4. So the latents come out 4x4 instead of 8x8, and the decoded image is half the size. With the real model the vae is 8, so it doesn't show up there. A fallback like this was added on purpose for Anima in Fix Anima text encoder exports and standalone core denoising #14943 so the block can run without a vae, but it gives the wrong size when the vae is not 8.
  7. charchit7 commented on Nov 19, 2026

    @charchit7
    Contributor

    @sayakpaul what do you think, Please correct me if I am missing something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    idea-contribution-welcomeWe want discussion, not a PR. feedbacks that help us will get co-author credit

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions