Skip to content

fix: follow the API's main-content default and send crawl --delay in seconds - #354

Open
rakshith48 wants to merge 2 commits into
mainfrom
capy/cli-api-defaults
Open

rakshith48 wants to merge 2 commits into
mainfrom
capy/cli-api-defaults

Conversation

@rakshith48

Copy link
Copy Markdown
Contributor

Two CLI defaults disagreed with the API.

Crawl --delay is sent in seconds. --delay <ms> is documented in milliseconds, but its value went to the API unchanged, and the API reads delay as seconds (at most 60). So --delay 1000 asked for 1000 seconds and was refused. The flag keeps its millisecond unit, and the CLI now divides by 1000 before sending: --delay 1500 sends delay: 1.5.

Scrapes follow the API's main-content default. scrape, download, batch-scrape and parse always sent onlyMainContent: false, so their output kept navigation, headers and footers, unlike the API, the SDKs and search --scrape.

  • Now these commands send onlyMainContent only when a flag sets it, so by default the API returns the main content.
  • A new --full-page flag sends onlyMainContent: false for the old output.
  • --only-main-content still works.
  • Passing both flags is refused.
  • The download wizard's "Only main content?" prompt now defaults to yes, and answering no sends false.

Visible change, for the next release notes. firecrawl scrape, download, batch-scrape and parse now return the page's main content by default, without navigation, headers or footers. Add --full-page to keep the whole page as before. crawl --delay now works as documented, in milliseconds.

This is meant to ship in the release after 1.26.24. The README option tables mention --full-page.

Tests. New tests cover:

  • the unset default, --full-page, and both flags together;
  • batch-scrape with --full-page;
  • the delay conversion.

The batch-scrape and crawl tests that asserted the old request bodies now assert the new ones. I also checked the built CLI against a local capture server: plain scrape sends no onlyMainContent, --full-page sends false, --only-main-content sends true, and crawl --delay 1500 sends delay: 1.5. Type check and build pass, and so does the full suite (877 tests).

Open in Capy

…seconds

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread src/index.ts
… calls too

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would require human review. Fixes CLI/API mismatches for crawl --delay and scrape-family main-content defaults, but the default-output change is user-visible and --full-page doesn't suppress the interactive wizard; a human should decide before shipping.

View guided diff | Turn on auto-fix | Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant