Repository navigation
perf: speed up _unescape_special_chars for large documents - #748
Open
SwastikTripathi wants to merge 1 commit into
Open
SwastikTripathi wants to merge 1 commit into
SwastikTripathi wants to merge 1 commit into
Conversation
Two O(n) costs were paid per pass through the while loop: 1. The termination check compared the full text string (`text == orig_text`), which is O(n) even when nothing changed. 2. The replacements dict was built from three intermediate tuples before every call (escape_table + code_table + reversed html_blocks). Fix both: - Skip immediately if 'md5-' is not present in the text (the common case for plain paragraphs with no inline code, links, or escapes). - Build the replacements dict by iterating the source dicts directly, without allocating intermediate tuples. - Replace the string comparison with a boolean flag set inside the re.sub callback; the loop breaks as soon as a full pass produces no replacements. Fixes trentm#635 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #635.
Problem
_unescape_special_charshas two hidden O(n) costs that compound on large markdown documents:Full-string comparison for loop termination. Each pass through the
while Trueloop storesorig_text = textand then comparestext == orig_textto detect whether anything changed. On a large document this is an O(n) string comparison even when the very first regex scan found nothing to replace.Unnecessary intermediate allocations on every call. The replacements dict was built from three
tuple(...)calls, converting_escape_table.items(),_code_table.items(), and a generator expression overhtml_blocks.items(), before iterating them into the dict. That's three extra list/tuple objects per call.Fix
'md5-'is not present anywhere intext, return immediately. This is the common case for plain paragraphs that contain no inline code, links, backslash escapes, or raw HTML, skipping both the dict build and the regex engine entirely.replacedflag (a one-element list so the inner function can mutate it) is set inside there.subcallback whenever a match is found in the lookup table. The loop breaks when a full pass sets no flag, no full-string comparison needed.All existing tests pass (
test/test.py,test/api.doctests).🤖 Generated with Claude Code