You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Multi-vector textual inversion uses only the first vector when the token has uppercase letters (transformers v5) #15313
With transformers v5, a multi-vector textual inversion embedding only contributes its first vector when its token contains uppercase letters. This happens both when the token is loaded that way (token="EasyNegative") and when it is written that way in the prompt (Neg for a token loaded as neg). No error or warning is raised.
maybe_convert_prompt should expand the token to <token> <token>_1 ... <token>_n before tokenizing. In v5, CLIPTokenizer matches added tokens case-insensitively and tokenize() returns them lowercased. But _maybe_convert_prompt looks the result up in tokenizer.added_tokens_encoder and replaces it in the prompt case-sensitively, so nothing gets expanded. With transformers 4.57.1, tokenize() returned the token as registered and the expansion worked.
Proposed fix: in _maybe_convert_prompt, find the registered token through its lowercased form and replace it case-insensitively. Happy to open a PR with a test if this approach works for you.
Reproduction
importtorchfromdiffusersimportStableDiffusionPipelinepipe=StableDiffusionPipeline.from_pretrained("hf-internal-testing/tiny-sd-pipe", safety_checker=None)
dim=pipe.text_encoder.config.hidden_sizepipe.load_textual_inversion(torch.randn(3, dim), token="MyToken") # adds MyToken, MyToken_1, MyToken_2pipe.load_textual_inversion(torch.randn(3, dim), token="neg") # adds neg, neg_1, neg_2fortextin ["MyToken", "neg", "Neg"]:
prompt=pipe.maybe_convert_prompt(text, pipe.tokenizer) # what the pipeline passes to the tokenizerprint(f"{text!r:10} -> {pipe.tokenizer.tokenize(prompt)}")
Expected: all three vectors in every case, e.g. 'MyToken' -> ['mytoken', 'mytoken_1', 'mytoken_2'].
Context
This affects embeddings made for A1111, which are referred to by their capitalized file name in prompts. For example, EasyNegative has 8 vectors. With the SD 1.5 tokenizer and the token loaded as EasyNegative, the prompt EasyNegative, blurry is expanded to 8 tokens with transformers 4.57.1 and stays a single token with 5.22.2.
Activity