docs: add guide on building custom extensions

[shixi-li]

Description

Refs #1936. Adds docs/guides/extending_crawlee.mdx, a page that maps Crawlee's extension points.

The four extension points you listed in the issue each already document their own contract in the guide that owns them: crawlers in the HTTP crawlers guide, HTTP clients and storage clients in theirs, and browser plugins in the Playwright crawler guide (#2089). What is still missing is the map, so someone who wants to extend Crawlee has to already know which guide to open, and a third-party project has no single page to point its users at for "here is the interface this integration implements".

This page is that map. For each extension point it states when to subclass rather than configure, names the base class and what its contract covers, and links to the guide that goes deep. It deliberately does not restate those guides.

Contents

• Extension points — a class diagram plus one section each for crawlers (AbstractHttpCrawler), HTTP clients (HttpClient), storage clients (StorageClient), and browser plugins (PlaywrightBrowserPlugin), each linking to its detailed guide.
• Choosing an extension point — matches the layer to what actually differs in an integration, and says to prefer configuration over a subclass where the built-in class already exposes the knob.
• Framing for third-party integrations, per the issue's rationale: they host their own guide and reference the contract here.

Notes

• Placed as its own guide rather than a section inside an existing one, since it spans all four extension points. The sidebar picks it up automatically (autogenerated over docs/guides).
• No new code examples: each linked guide already carries a runnable one for its own extension point, so duplicating them here would mean two copies to keep in sync.
• Happy to adjust the placement, depth, or wording, especially if you would rather this live under a different section or carry a worked example of its own.

Testing

• uv run poe build-docs — build succeeds, the page renders at /docs/next/guides/extending-crawlee, and the build reports no broken links or anchors for it (the broken anchors in the log are pre-existing ones on API pages).
• Verified every link target resolves before building: the five doc ids (http-crawlers, http-clients, storage-clients, playwright-crawler, architecture-overview), the playwright-crawler-with-camoufox example, and each ApiLink class (AbstractHttpCrawler, BasicCrawler, HttpClient, StorageClient, PlaywrightBrowserPlugin, PlaywrightBrowserController, BrowserPool) against src/crawlee/.

AI assistance

Written with AI assistance (Claude). The scoping decision was mine to check first what each extension point already documents, which is why this is a map rather than four new sections; the browser-plugin quarter was covered by my earlier #2089.

[Copilot]

Pull request overview

Adds a new documentation guide that serves as a “map” of Crawlee’s primary extension points (crawlers, HTTP clients, storage clients, and browser plugins), briefly describing when to subclass vs configure and linking to the deeper, existing guides for each contract.

Changes:

• Add docs/guides/extending_crawlee.mdx with an overview of extension points and links to the relevant detailed guides/examples.
• Include a high-level Mermaid class diagram summarizing the main extensibility surfaces.
• Add a short “Choosing an extension point” decision checklist to help integrators pick the right layer.

Suppressed comments (1)

docs/guides/extending_crawlee.mdx:73

• This section describes PlaywrightBrowserPlugin as the base for browser plugins, but the abstract contract is BrowserPlugin (with PlaywrightBrowserPlugin being the built-in Playwright implementation). Updating the wording avoids confusion about which class defines the core interface vs which class is the default implementation to configure/subclass.

Subclass a browser plugin when an integration launches browsers through an API other than the standard Playwright one. Configuration options on <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink> cover the cases where the standard launch API is enough, so reach for a subclass only when the launch path itself differs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Mantisus]

Thanks for the PR and the contribution. The direction is right, but the page needs further refinement.

In its current state the guide doesn't match the documentation style of the project, and that's what most of my comments are about.

A few things that run across the page rather than sitting in one place:

• Sections open with "subclass X when...", so they assume the reader already knows what the component is. Our guides describe the component first, then get to the details.
• Class names are given in prose instead of ApiLink, which we use everywhere else.
• The contracts aren't named anywhere, even though the page description promises them.
• Style details: we use contractions, we don't glue independent clauses with a semicolon, and we avoid both repeating the same opening across parallel list items and using bold to carry meaning.

The rest is inline. I think this needs another pass over the structure rather than line edits, happy to take another look once it's reworked.

[shixi-li]

Thanks for the detailed review — that was exactly the kind of feedback the page needed. Reworked the structure rather than line-editing, as you suggested. Every section now opens with what the component is and what it does, then names its contract, then gets to what you subclass.

Point by point:

Intro (L9/L11). Now leads with the motivation — the cases where Crawlee doesn't do what you need and forking is the wrong answer — before any mechanics. Dropped the possessive pile-up and the "stable" promise; the third-party paragraph is one sentence and no longer mixes "you" and "your users".

"Four extension points" (L15). Reworded to say these are the main ones and explicitly names RequestLoader, FingerprintGenerator, and RenderingTypePredictor as also extensible.

The diagram (L51). Redrawn to show what the sections actually describe: AbstractHttpCrawler and PlaywrightCrawler under BasicCrawler, the three per-storage clients StorageClient opens, and the BrowserController that new_browser() returns. The extensible classes carry <<abstract>>, and the contract methods that matter (create_dataset_client / create_kvs_client / create_rq_client, new_browser) are on the boxes, so the diagram now says something about extensibility rather than just relations.

Crawlers (L59). Starts from BasicCrawler and what it owns (queue, fetch, context, retries, concurrency, sessions, storage) and why that makes it the base. The parser contract is named method by method: parse, parse_text, select, is_matching_selector, find_links.

Storage clients (L75). Starts with what a storage client is and why swapping it is useful. StorageClient is described as the three factory methods it is, and DatasetClient, KeyValueStoreClient, RequestQueueClient now have ApiLinks plus a line each on what they actually do, since that's where the work is. Built-in clients are ApiLinks throughout.

Contracts. Each section now names its contract explicitly — that was promised in the page description and missing before.

Style. Class names go through ApiLink everywhere. Contractions used. No semicolons joining independent clauses. No bold carrying meaning. The "Choosing an extension point" list items no longer share an opening — each starts from the thing that differs in your integration.

Verified locally: all 23 ApiLink targets resolve to real classes in src/crawlee, the Mermaid diagram renders, and every internal doc link points at an existing page.

[Mantisus]

Thanks for the rework, this is a big step up.

One thing that runs across the page. We don't use a spaced hyphen to join clauses. We split them into separate sentences instead.

[shixi-li]

@Mantisus, thanks for the second pass. I’ve rebased onto the latest master and pushed 1c16751. The update redesigns the diagram around the actual extension contracts, adds the browser-crawler path, distinguishes BrowserPlugin from the common PlaywrightBrowserPlugin path, restores configure-before-subclass guidance, makes the checklist parallel, and applies the contract-stability, closing, and style changes.

Validation is green:

• uv run poe build-docs
• pnpm lint:md
• pnpm format:check

I’ve replied to each of the nine actionable threads. Could you take another look?

[Mantisus]

Thanks for the update. It looks good!