Skip to content

FIX: decode variation selector payloads next to emoji with presentation selectors - #3141

Open
Jaafer Rahmani (bIackr0se) wants to merge 1 commit into
microsoft:mainfrom
bIackr0se:fix/variation-selector-decode-anchor
Open

Jaafer Rahmani (bIackr0se) wants to merge 1 commit into
microsoft:mainfrom
bIackr0se:fix/variation-selector-decode-anchor

Conversation

@bIackr0se

Copy link
Copy Markdown
Contributor

Description

VariationSelectorSmugglerConverter(action="decode") returns '\x0f' instead of the hidden text when an ordinary emoji such as ❤️ or ✔️ comes before the payload. These emoji are written as the symbol plus U+FE0F, a variation selector that picks the emoji presentation. The decoder took the first variation selector anywhere in the text as the start of the payload, decoded that U+FE0F as byte 0x0F and stopped at the next ordinary character.

decode_visible_hidden had a related problem: it split at the first base character (😊 by default), so visible text that contained 😊 was cut short.

The payload secret, encoded with the default settings and placed after different carrier text:

                                              main                  this PR
decode  "Sure! " + payload                    'secret'              'secret'
decode  "I ❤️ this " + payload                '\x0f'                'secret'
decode  "Done ✔️ " + payload                  '\x0f'                'secret'
decode_visible_hidden("Hi 😊 there ", ...)    ('Hi ', 'secret')     ('Hi 😊 there ', 'secret')
decode_visible_hidden("Hi 😊❤️ there ", ...)  ('Hi ', '\x0f')       ('Hi 😊❤️ there ', 'secret')

The results are the same with embed_in_base=False.

The fix:

  • decode_message starts at the variation selectors that follow the converter's base character, with or without the space separator. Text that has no such payload is scanned as before, so a payload encoded with a different base character still decodes.
  • decode_visible_hidden splits at the last base character, which is the one encode_visible_hidden appends after the visible text (with a space as the base character and embed_in_base=False, the space before the separator). The payload holds only variation selectors, so it cannot contain the base character.
  • The constructor and the new payload search share one variation selector range check.

Tests and Documentation

  • test_token_smuggler_converter.py: decoding with ❤️, ✔️ and 😊 before the payload, in both embedding modes; visible/hidden round trips with 😊 and ❤️ in the visible text and with an empty hidden text; decoding with the other embed_in_base setting; a space as the base character with the separator; decoding a payload encoded with another base character. 20 of the new cases fail on main.
  • tests/unit/converter passes (2094 passed, 34 skipped). ruff is clean, and ty (0.0.84) reports the same diagnostics as on main.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant