Skip to content

Add SE5 Fix encoding errors plugin - #321

Merged
niksedk merged 1 commit into
mainfrom
claude/se5-fix-mojibake
Oct 13, 2026
Merged

niksedk merged 1 commit into
mainfrom
claude/se5-fix-mojibake

Conversation

@niksedk

@niksedk niksedk commented Oct 13, 2026

Copy link
Copy Markdown
Member

New SE5 plugin Fix encoding errors (se5/FixEncodingErrors). It repairs UTF-8 text that was read with the wrong code page and saved again (mojibake):

  • résumé → résumé, don’t → don’t, 25° → 25°, 😀 → 😀
  • rã©sumã© → résumé when the text was lowercased after it broke (a user report: Madam Secretary SRTs extracted from MKV)
  • Привет / Привет → Привет, and Greek, Hebrew and Arabic misread as Western
  • lost 0x9D (†→ ”), no-break space turned into a normal space (à → à), BOM as text (), text broken twice

Matroska text tracks are UTF-8 by spec, so SE shows what is stored; the damage happened before muxing.

How it works: each character is mapped back to its byte in the code page and runs that form valid UTF-8 are decoded. Code pages: 1252, 1250, 1251, 1253, 1254 and 1257, with auto-detect.

False-fix guards:

  • A decode must land in a plausible script, so real Macedonian ој never becomes Greek μ.
  • Ambiguous matches (Não”, irmã…) are fixed only when the subtitle is clearly broken: it has unambiguous matches and they cover ≥30% of its non-ASCII characters.

Checked:

  • 43 unit tests.
  • False-positive sweep over all SE UI translations and word lists (about 359k lines, 35 languages). With auto-detect the only change is one line that really is broken.
  • Round-trip: every language file breaks and fixes back at 100% through its natural code page.

UI follows Italic each line: a code-page picker, All/Selected lines scope, and checkable before/after cards with diff highlighting. Light and dark were checked headless. Not yet run inside a real SE.

screenshot

After merge: dispatch fix-encoding-errors.yml with release=true, then add it to se5-plugins.json.

🤖 Generated with Claude Code

Repairs UTF-8 text that was read with the wrong code page (mojibake):
résumé -> résumé, don’t -> don’t, also when lowercased afterwards
(rã©sumã©). Western, Central European, Cyrillic, Greek, Turkish and
Baltic code pages with auto-detect; checkable before/after preview.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@niksedk
niksedk merged commit c45e0a6 into main Oct 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant