-
Notifications
You must be signed in to change notification settings - Fork 525
Add Puppeteer support for Worker JavaScript #148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
scuffi
wants to merge
1
commit into
main
Choose a base branch
from
computer/puppeteer-plugin
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+3,605
−32
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "@cloudflare/computer": minor | ||
| --- | ||
|
|
||
| Add Worker JavaScript plugins and an opt-in Puppeteer plugin backed by a Browser Run binding. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| # Browser automation in Worker JavaScript | ||
|
|
||
| `@cloudflare/computer/plugins/puppeteer` lets code running in `WorkerJavaScriptBackend` use Cloudflare Browser Run. Puppeteer and its `Browser` and `Page` objects stay inside the isolated Dynamic Worker; Chromium runs in Browser Run. | ||
|
|
||
| ## Configure the plugin | ||
|
|
||
| The host Worker needs Worker Loader and Browser Run bindings: | ||
|
|
||
| ```jsonc | ||
| { | ||
| "compatibility_flags": ["nodejs_compat", "experimental"], | ||
| "worker_loaders": [{ "binding": "LOADER" }], | ||
| "browser": { "binding": "BROWSER" } | ||
| } | ||
| ``` | ||
|
|
||
| Pass both bindings to the backend: | ||
|
|
||
| ```ts | ||
| import { DurableObject } from "cloudflare:workers"; | ||
| import { type DurableObjectStorageLike, Workspace } from "@cloudflare/computer"; | ||
| import { WorkerJavaScriptBackend } from "@cloudflare/computer/backends/worker-javascript"; | ||
| import { puppeteer } from "@cloudflare/computer/plugins/puppeteer"; | ||
|
|
||
| export class BrowserWorkspace extends DurableObject<Env> { | ||
| readonly workspace: Workspace; | ||
|
|
||
| constructor(ctx: DurableObjectState, env: Env) { | ||
| super(ctx, env); | ||
| this.workspace = new Workspace({ | ||
| storage: ctx.storage as unknown as DurableObjectStorageLike, | ||
| backends: [ | ||
| new WorkerJavaScriptBackend({ | ||
| loader: env.LOADER, | ||
| plugins: [puppeteer({ browser: env.BROWSER })], | ||
| }), | ||
| ], | ||
| }); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The plugin bundles the Worker-compatible Puppeteer client, so the application does not need a separate runtime dependency on `@cloudflare/puppeteer`. | ||
|
|
||
| ## Run a browser task | ||
|
|
||
| Code passed to `workspace.runtime.exec()` imports the configured module normally: | ||
|
|
||
| ```ts | ||
| using execution = await workspace.runtime.exec( | ||
| ` | ||
| import { withBrowser } from "@cloudflare/puppeteer"; | ||
|
|
||
| export default (input) => withBrowser(async (browser) => { | ||
| const page = await browser.newPage(); | ||
| await page.goto(input.url, { waitUntil: "domcontentloaded" }); | ||
| return { title: await page.title(), finalUrl: page.url() }; | ||
| }, { | ||
| guardrails: { allowedDomains: [input.hostname] }, | ||
| }); | ||
| `, | ||
| { | ||
| input: { | ||
| url: "https://developers.cloudflare.com/agents/", | ||
| hostname: "developers.cloudflare.com", | ||
| }, | ||
| }, | ||
| ); | ||
|
|
||
| const result = await execution.result(); | ||
| ``` | ||
|
|
||
| `withBrowser(callback, options?)` launches a connection-bound browser, runs the callback, and closes the browser afterward. Use `launch(options?)` when code needs to manage the browser itself: | ||
|
|
||
| ```js | ||
| import { launch } from "@cloudflare/puppeteer"; | ||
|
|
||
| const browser = await launch(); | ||
| try { | ||
| // Use Puppeteer normally. | ||
| } finally { | ||
| await browser.close(); | ||
| } | ||
| ``` | ||
|
|
||
| The module also exports `browserBinding`, the unchanged upstream default export, and upstream runtime exports. `browserBinding` is useful for APIs such as `puppeteer.sessions()` that take the Browser Run binding directly. | ||
|
|
||
| Do not return Puppeteer objects from an execution. Return structured data or write larger output to the Workspace. | ||
|
|
||
| ## Save browser output | ||
|
|
||
| Worker JavaScript provides Workspace-backed `node:fs` and `node:fs/promises`. A screenshot can be written without returning its bytes through the structured result: | ||
|
|
||
| ```js | ||
| import { withBrowser } from "@cloudflare/puppeteer"; | ||
| import fs from "node:fs/promises"; | ||
|
|
||
| export default (input) => withBrowser(async (browser) => { | ||
| const page = await browser.newPage(); | ||
| await page.goto(input.url); | ||
| await fs.writeFile(input.outputPath, await page.screenshot({ type: "png" })); | ||
| return { title: await page.title(), outputPath: input.outputPath }; | ||
| }); | ||
| ``` | ||
|
|
||
| The Dynamic Worker is disposable, but files written to the Workspace remain available to later executions. | ||
|
|
||
| ## Authority and limits | ||
|
|
||
| Installing the plugin grants every execution on that backend access to its public browser API. Put browser-enabled work on a separate named backend when only some callers should have that authority. Plugins installed on one backend are mutually trusted and share the plugin binding authority domain; caller modules and ordinary configured modules cannot import the internal binding bridge. | ||
|
|
||
| Browser navigation happens through Browser Run, not through the backend's `globalOutbound` policy. Validate user input and set Browser Run guardrails when the application accepts URLs from other users. | ||
|
|
||
| Computer execution timeouts and Puppeteer navigation timeouts are separate. Set both for the workload. `withBrowser()` closes the browser after normal completion or an error. Cancellation and timeout dispose the Dynamic Worker and its client connection, so application cleanup code may not finish in those paths. | ||
|
|
||
| Screenshots are encoded when they cross the Workspace filesystem bridge. For larger screenshots, raise the capability limits deliberately: | ||
|
|
||
| ```ts | ||
| new WorkerJavaScriptBackend({ | ||
| loader: env.LOADER, | ||
| plugins: [puppeteer({ browser: env.BROWSER })], | ||
| maxCapabilityBytes: 8 * 1024 * 1024, | ||
| maxCapabilityRequestBytes: 16 * 1024 * 1024, | ||
| }); | ||
| ``` | ||
|
|
||
| See [`examples/browser-rendering`](../examples/browser-rendering) for a complete self-hosted example that scrapes pages and writes Markdown, JSON, and PNG output to a durable Workspace. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| dist/ | ||
| node_modules/ | ||
| .wrangler/ | ||
| worker-configuration.d.ts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # Computer browser rendering example | ||
|
|
||
| This is a self-hosted demonstration of the public `@cloudflare/computer/plugins/puppeteer` integration. A generated ECMAScript module gets the normal Puppeteer `Browser` and `Page` APIs, while Cloudflare Browser Run supplies the managed Chromium session. | ||
|
|
||
| ## Plugin usage | ||
|
|
||
| The reusable integration has two pieces. First, register the plugin on a Worker JavaScript backend: | ||
|
|
||
| ```ts | ||
| import { Workspace } from "@cloudflare/computer"; | ||
| import { WorkerJavaScriptBackend } from "@cloudflare/computer/backends/worker-javascript"; | ||
| import { puppeteer } from "@cloudflare/computer/plugins/puppeteer"; | ||
|
|
||
| const workspace = new Workspace({ | ||
| storage: ctx.storage, | ||
| backends: [ | ||
| new WorkerJavaScriptBackend({ | ||
| loader: env.LOADER, | ||
| plugins: [puppeteer({ browser: env.BROWSER })], | ||
| }), | ||
| ], | ||
| }); | ||
| ``` | ||
|
|
||
| Then use the bound lifecycle helper inside an ordinary `workspace.runtime.exec()` module: | ||
|
|
||
| ```js | ||
| import { withBrowser } from "@cloudflare/puppeteer"; | ||
|
|
||
| export default (input) => withBrowser(async (browser) => { | ||
| const page = await browser.newPage(); | ||
| await page.goto(input.url); | ||
| return { title: await page.title() }; | ||
| }, { | ||
| guardrails: { allowedDomains: [input.hostname] }, | ||
| }); | ||
| ``` | ||
|
|
||
| That is the complete plugin boundary. `withBrowser()` uses the configured Browser Run binding and closes the connection-bound browser after the callback. `Browser`, `Page`, selectors, and page evaluation stay inside the Dynamic Worker; Chromium runs in Browser Run. | ||
|
|
||
| ## What the example adds | ||
|
|
||
| The rest of this directory is an example application, not code required by the plugin. Its web interface accepts a user-provided HTTP(S) URL and adds: | ||
|
|
||
| - text, heading, metadata, and link scraping; | ||
| - a full-page screenshot written to the durable Workspace; | ||
| - response, document, viewport, and navigation timing data; | ||
| - a research workflow that writes `report.md`, `page.json`, and `screenshot.png` to one durable Workspace directory; | ||
| - artifact routes, result components, and a Workspace file tree. | ||
|
|
||
| Those pieces show ways to combine Browser Run with Computer's durable filesystem. Applications can instead execute a module as small as the one above. | ||
|
|
||
| ## Run it | ||
|
|
||
| From the repository root: | ||
|
|
||
| ```sh | ||
| npm install | ||
| npm run build --workspace @cloudflare/computer | ||
| npm run dev --workspace @example/computer-browser-rendering | ||
| ``` | ||
|
|
||
| Open the URL printed by Wrangler. Local development uses the Browser Run binding, so requests consume Browser Run quota and need a Cloudflare account with Browser Run access. | ||
|
|
||
| Local development does not require authentication. Before deploying, set a token and deploy from the example directory: | ||
|
|
||
| ```sh | ||
| cd examples/browser-rendering | ||
| npx wrangler secret put DEMO_TOKEN | ||
| npm run deploy | ||
| ``` | ||
|
|
||
| The deployed site uses HTTP Basic authentication. Enter `demo` as the username and the secret as the password. | ||
|
|
||
| The Worker needs both bindings shown in `wrangler.jsonc`: | ||
|
|
||
| ```jsonc | ||
| { | ||
| "compatibility_flags": ["nodejs_compat", "experimental"], | ||
| "worker_loaders": [{ "binding": "LOADER" }], | ||
| "browser": { "binding": "BROWSER" } | ||
| } | ||
| ``` | ||
|
|
||
| ## Files | ||
|
|
||
| - `src/index.ts` configures Computer and exposes the API and durable artifact routes. | ||
| - `src/execution-source.ts` defines the browser task that runs inside the Dynamic Worker and writes the research bundle through Computer's built-in `node:fs/promises` module. | ||
| - `src/ui.ts` contains the dependency-free demonstration interface. | ||
| - `wrangler.jsonc` declares the Worker Loader, Browser Run, and Durable Object bindings. | ||
|
|
||
| The example requires authentication when deployed, applies Browser Run guardrails for the requested host, and uses connection-bound browser sessions. Add workload-specific URL policy and quota handling if you adapt it for a shared service. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "name": "@example/computer-browser-rendering", | ||
| "version": "0.0.0", | ||
| "private": true, | ||
| "type": "module", | ||
| "description": "Browser Run demo that executes Cloudflare Puppeteer inside the Computer Worker JavaScript backend.", | ||
| "scripts": { | ||
| "dev": "wrangler dev", | ||
| "deploy": "wrangler deploy", | ||
| "build:types": "wrangler types", | ||
| "test": "vitest run", | ||
| "typecheck": "tsc --noEmit" | ||
| }, | ||
| "dependencies": { | ||
| "@cloudflare/computer": "*" | ||
| }, | ||
| "devDependencies": { | ||
| "@cloudflare/workers-types": "^4.20260616.1", | ||
| "typescript": "^6.0.3", | ||
| "vitest": "^4.1.11", | ||
| "wrangler": "^4.130.0" | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.