Skip to content

Precompute block line markers per line, not per character - #449

Open
John-Cusack wants to merge 2 commits into
executablebooks:masterfrom
John-Cusack:rs-upstream-perf
Open

John-Cusack wants to merge 2 commits into
executablebooks:masterfrom
John-Cusack:rs-upstream-perf

Conversation

@John-Cusack

Copy link
Copy Markdown

Summary

StateBlock.__init__ built its line markers (bMarks/eMarks/tShift/sCount/bsCount) by dispatching every source character through a Python-level loop (enumerate + isStrSpace call per char, plus a branch per char for the rest of the line). This patch builds the identical markers once per line: str.find locates each \n at C speed and only leading whitespace is scanned (usually zero iterations). The scan body (space/tab counting with tab-stop expansion) is unchanged, so the recorded markers are exactly the same.

Single file changed: markdown_it/rules_block/state_block.py (+29/-29). No parsing logic touched, no new dependency, no behavior change.

Correctness

  • Full suite on this branch: 1031 passed, 1 skipped — identical counts to master (6f58654) in the same environment.
  • Render differential (base vs patched): in-repo tests/ fixtures aside, rendered output compared over a 16.7KB mixed document, a 280KB bulk document, and 30 edge inputs (empty, lone \n, CRLF, lone CR, tabs, deep indents, 10KB single line, 20K-line list, entities, emoji, tables, fences) × default/commonmark/zero presets — 0 divergences.
  • ruff check and black --check clean on the changed file.

Performance

Frozen workload below (self-contained repro script, stdlib + installed markdown-it-py only, seed-fixed; run it under each tree). Best of 6 interleaved fresh-process rounds, CPython 3.12.3, Linux:

Workload Base (master) Patched Delta
15 renders, ~13KB synthetic mixed doc (784 lines), median 79.7 ms 65.7 ms 1.21x
Same, best single round 76.3 ms 60.2 ms 1.27x
16.7KB mixed doc, per render 4.55 ms 3.75 ms 1.20x
280KB bulk doc, per render 76.4 ms 66.9 ms 1.14x
Single 100KB line, per render (pathological case for the old loop) 5.42 ms 0.93 ms 5.8x
20K short lines / empty input faster / flat faster / flat no regressions

Memory (tracemalloc peak over the repro workload): 1200469 B on both sides — neutral, as expected (same lists, same lengths).

Reproduction

import os
import random
import statistics
import time
import tracemalloc

try:
    os.sched_setaffinity(0, {3} if os.cpu_count() > 3 else {0})
except (AttributeError, OSError):
    pass

from markdown_it import MarkdownIt

rng = random.Random(20261014)
BLOCKS = [
    "# Heading {i}\n",
    "## Sub heading {i}\n",
    "Paragraph {i} with *emphasis*, **strong**, `code` and a [link](https://example.com/{i}).\n",
    "- item {i}a\n- item {i}b\n  - nested {i}\n",
    "> quote {i}\n> > nested {i}\n",
    "```python\ndef f{i}():\n    return {i}\n```\n",
    "1. first {i}\n2. second {i}\n",
    "| a{i} | b{i} |\n|---|---|\n| 1 | 2 |\n",
    "    indented code {i}\n",
    "***\n",
]
doc = "".join(rng.choice(BLOCKS).format(i=i) for i in range(400))
print(f"doc_bytes={len(doc.encode())} lines={doc.count(chr(10))}")

md = MarkdownIt()
out = md.render(doc)
assert len(out) > len(doc) // 2 and "<" in out
for _ in range(3):
    md.render(doc)

walls = []
tracemalloc.start()
for _ in range(15):
    t0 = time.perf_counter_ns()
    md.render(doc)
    walls.append((time.perf_counter_ns() - t0) / 1e6)
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
walls.sort()
print(f"median_ms={statistics.median(walls):.3f} min_ms={walls[0]:.3f} peak_B={peak}")

Checklist

  • Tests pass (pytest tests/: 1031 passed, 1 skipped, same as master)
  • Code style (ruff check, black --check clean)
  • No behavior change (render differential: 0 divergences)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant