Skip to content

fix: reject lookalike canonical documentation URLs - #2

Open
rudycelekli wants to merge 2 commits into
firecrawl:mainfrom
rudycelekli:fix/docs-canonical-boundaries
Open

rudycelekli wants to merge 2 commits into
firecrawl:mainfrom
rudycelekli:fix/docs-canonical-boundaries

Conversation

@rudycelekli

Copy link
Copy Markdown

The canonical docs matcher uses a raw prefix check, so docs.example.com.evil.test/guide scores as docs.example.com/guide, and /docs-other/guide as /docs/guide. Require a host/base-path boundary while preserving existing URL normalization and page-stem rules.

Adds 16 offline cases, including report-generation coverage where a canonical URL plus a lookalike incorrectly scores 1.0 instead of 0.5. Seven cases fail on exact upstream; all 148 tests pass on Python 3.11 with the patch. Base-to-head diff check passes.

This corrects scoring behavior. Published tables remain untouched; maintainers should rescore saved records before updating affected numbers. No provider calls were made.

AI-assisted contribution; code and regression evidence reviewed before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant