Skip to content

Multi-vector textual inversion uses only the first vector when the token has uppercase letters (transformers v5) #15313

Open

Description

@sohailshk

Describe the bug

With transformers v5, a multi-vector textual inversion embedding only contributes its first vector when its token contains uppercase letters. This happens both when the token is loaded that way (token="EasyNegative") and when it is written that way in the prompt (Neg for a token loaded as neg). No error or warning is raised.

maybe_convert_prompt should expand the token to <token> <token>_1 ... <token>_n before tokenizing. In v5, CLIPTokenizer matches added tokens case-insensitively and tokenize() returns them lowercased. But _maybe_convert_prompt looks the result up in tokenizer.added_tokens_encoder and replaces it in the prompt case-sensitively, so nothing gets expanded. With transformers 4.57.1, tokenize() returned the token as registered and the expansion worked.

Proposed fix: in _maybe_convert_prompt, find the registered token through its lowercased form and replace it case-insensitively. Happy to open a PR with a test if this approach works for you.

Reproduction

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("hf-internal-testing/tiny-sd-pipe", safety_checker=None)
dim = pipe.text_encoder.config.hidden_size
pipe.load_textual_inversion(torch.randn(3, dim), token="MyToken")  # adds MyToken, MyToken_1, MyToken_2
pipe.load_textual_inversion(torch.randn(3, dim), token="neg")  # adds neg, neg_1, neg_2

for text in ["MyToken", "neg", "Neg"]:
    prompt = pipe.maybe_convert_prompt(text, pipe.tokenizer)  # what the pipeline passes to the tokenizer
    print(f"{text!r:10} -> {pipe.tokenizer.tokenize(prompt)}")

Output on main:

'MyToken'  -> ['mytoken']
'neg'      -> ['neg', 'neg_1', 'neg_2']
'Neg'      -> ['neg']

Expected: all three vectors in every case, e.g. 'MyToken' -> ['mytoken', 'mytoken_1', 'mytoken_2'].

Context

This affects embeddings made for A1111, which are referred to by their capitalized file name in prompts. For example, EasyNegative has 8 vectors. With the SD 1.5 tokenizer and the token loaded as EasyNegative, the prompt EasyNegative, blurry is expanded to 8 tokens with transformers 4.57.1 and stays a single token with 5.22.2.

Logs

No error or warning is raised.

System Info

  • 🤗 Diffusers version: 0.44.0.dev0 (main, 1dabc29)
  • Platform: macOS-26.6.2-arm64-arm-64bit
  • Running on Google Colab?: No
  • Python version: 3.12.11
  • PyTorch version (GPU?): 2.15.0 (False)
  • Huggingface_hub version: 2.4.0
  • Transformers version: 5.22.2
  • Accelerate version: 1.15.0
  • PEFT version: 0.21.2
  • Safetensors version: 0.8.0
  • xFormers version: not installed
  • Accelerator: Apple M5 Pro
  • Using GPU in script?: No
  • Using distributed or parallel set-up in script?: No

Who can help?

@sayakpaul

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions