Skip to content

fix: preserve path and query casing in normalize_url (#2008) - #2161

Draft
aryansk wants to merge 1 commit into
apify:masterfrom
aryansk:fix/2008-normalize-url-case
Draft

fix: preserve path and query casing in normalize_url (#2008)#2161
aryansk wants to merge 1 commit into
apify:masterfrom
aryansk:fix/2008-normalize-url-case

Conversation

@aryansk

@aryansk aryansk commented Aug 15, 2026

Copy link
Copy Markdown

Summary

Fixes #2008normalize_url lowercased the entire URL, including the path and query, so any two URLs differing only in path or query casing produced the same unique key:

  • https://example.com/Product/ABC and https://example.com/product/abc → same key
  • https://example.com/?token=SeCrEt and https://example.com/?token=secret → same key

Per RFC 3986 §6.2.2.1 only the scheme and host are case-insensitive — the path and query are case-sensitive. On sites with case-sensitive paths (base64/hashid identifiers, usernames), case-distinct pages were silently deduplicated with no log message or statistic revealing it.

Change

yarl already lowercases the scheme and host on parse (verified empirically: URL('HTTP://Example.COM/Product/ABC?Token=SeCrEt')http://example.com/Product/ABC?Token=SeCrEt). The fix removes the final .lower() on the constructed URL so path/query/fragment casing survives normalization, matching browser behavior. The docstring now states precisely which parts are case-folded.

Breaking change note: this changes how default unique keys are computed — crawls that relied on case-insensitive dedup will now visit more pages, and keys stored in persisted queues won't match newly computed ones. Per the issue, this should land in a major (2.0) release with the change documented loudly.

Validation

  • New parametrized cases: scheme/host still lowercased (HTTPS://EXAMPLE.COM/?KEY=VALUEhttps://example.com/?KEY=VALUE), path/query/fragment case preserved, trailing-slash removal unaffected.
  • New regression test: case-distinct paths normalize to different URLs (dedup no longer collapses them).
  • tests/unit/_utils — 299 passed, ruff check + format clean.

normalize_url lowercased the entire URL, including the path and query, so
any two URLs differing only in path or query casing produced the same
unique key and case-distinct pages were silently deduplicated. Per RFC 3986
only the scheme and host are case-insensitive; yarl already lowercases
those on parse, so the explicit `.lower()` is removed and path/query/
fragment casing is preserved. This is a breaking change for persisted
unique keys (issue suggests landing in 2.0).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

normalize_url lowercases the entire URL, silently deduplicating case-distinct pages

2 participants