[ybai08]

This fixes #192 behind an opt-in option, as discussed in the review. The default ids are unchanged.

With the header-ids extra, a markdown header's id is slugified from the raw markdown source, so link URLs, image paths and emphasis markers end up in the id. With the new strip-syntax option, extras={"header-ids": {"strip-syntax": True}}, the id is made from the rendered header instead:

Header	Default (unchanged)	strip-syntax
# [Inline link](https://github.com)	inline-linkhttpsgithubcom	inline-link
# [Reference link][ref]	reference-linkref	reference-link
#  Image	logologopng-image	image
# <span>Inline</span> HTML	spaninlinespan-html	inline-html
# __Strong__ and _em_	__strong__-and-_em_	strong-and-em

How it works with the option set:

• Id source: _h_sub builds the id from the rendered header, with tags stripped. The safe mode handling from Fix sanitized HTML leaking into header ids in safe mode #739 is kept: HTML spans are unhashed (to their sanitized form) before the id is built. In escape mode the escaped tags are visible text, so they still count towards the id.
• Footnotes: a footnote reference in a header is still a placeholder at this point (numbers are assigned after the block gamut), so the placeholder is dropped from the id text. Without that, the placeholder ended up in the id attribute, and _do_footnote_marker later failed with a ValueError. Footnoted headers keep the id they have by default (# Header with a footnote[^1] -> header-with-a-footnote1).
• Headers with only an HTML tag: a header whose only content is an HTML tag (# <sometext>) has no visible text, so it gets the same fallback id as other headers with no word characters (-1).

Without the option, nothing changes: header_id_from_text, the mixed-mode HTML headers and every default id are as on master. The module docstring describes the option, and the CHANGES entry is reworded to say it adds the option.

Tests:

• header_ids_issue192 (the headers above, plus a code span that keeps its underscores) and header_ids_footnote_ref (footnote refs in headers, which covers the ValueError above) now set strip-syntax. With lib/markdown2.py from master, both fail on the ids.
• header_ids_issue192_default and header_ids_footnote_ref_default render the same inputs without the option. Their expected HTML was generated with master's markdown2.py, so they pin the default ids.

Run on this head with the system Python 3.9.6: cd test && python test.py -- -knownfailure (280 tests, OK; the pygments, wavedrom, latex2mathml and emoji tests were skipped because those modules aren't installed in that Python), and python test/test_redos.py (10 tests, OK). The first version of this PR was also run on Python 3.12 with those modules installed. Not run on this head: pycodestyle (not installed here), the other Python versions, and the Windows and Linux CI jobs.

[Crozzers]

Have you measured perf on running span gamut on each header?

Plus this would break any existing hrefs if they were recompiled with this code

[ybai08]

Perf: the span gamut doesn't run an extra time. _h_sub already ran _run_span_gamut(header_group) once for the header's HTML; this moves that call above the id computation and builds the id from its result, so the only new work per header is one tag-stripping regex. Timing markdown() on master vs this branch (best of 7 × 5 runs, CPython 3.9.6, macOS arm64):

document	extras	master	branch
1000 headers with links/emphasis/code + paragraphs	header-ids	101.1 ms	100.9 ms
same	header-ids, toc	113.9 ms	114.3 ms
1000 plain headers	header-ids	22.2 ms	22.4 ms
same	header-ids, toc	23.7 ms	23.8 ms

Existing hrefs: ids stay the same for plain headers and for emphasis, code spans and entities (## *Emphasis* here → emphasis-here, ## Use `code` → use-code, ## Foo & Bar → foo-bar on both). They change only where the markdown source leaked into the id, i.e. headers with links or images: ## [Docs](http://x.io/a) goes from docshttpxioa to docs, and ## ![img](a.png) Logo from imgapng-logo to logo. So existing anchors to those headers would break when re-rendered.

If that's a concern, the new behaviour could sit behind a header-ids option (e.g. {"header-ids": {"from-rendered": True}}) with the current ids as the default. Happy to change it that way, or to close this if you'd rather keep ids as they are.

[nicholasserra]

@Crozzers Tell me more on your backwards compat concern. You're worried that someone who previously rendered header IDs and then put them online and shared links would have invalid links if they went and re-rendered this and published the page again?

[Crozzers]

Yeah that's pretty much it. I seem to remember previous changes having to be adapted around that compatibility concern.

Could wrap it behind an option on the extra as suggested, although not sure about the name from-rendered. Not very descriptive of what it does. Maybe something more like strip-syntax or something.

Then maybe at some point in the far future a 3.0.0 release can make this the default

[ybai08]

Makes sense. Done in e698958: the new behaviour is behind {"header-ids": {"strip-syntax": True}}, and without it every id is the same as on master.

• header_id_from_text is back to master's version. The tag stripping now happens only on the strip-syntax path, so the mixed-mode HTML headers and any subclass overriding header_id_from_text see no change by default.
• The two header_id_from_text used by extras "header-ids" doesn't parse markdown #192 cases set the option. Two _default copies of them, rendered with master's markdown2.py, pin the unchanged default ids.
• The docstring describes the option, and the CHANGES entry now says it adds strip-syntax.

I've updated the description to match. Happy to rename the option if you'd prefer something else.