Repository navigation
fix: follow the API's main-content default and send crawl --delay in seconds - #354
Open
rakshith48 wants to merge 2 commits into
Open
rakshith48 wants to merge 2 commits into
rakshith48 wants to merge 2 commits into
Conversation
…seconds Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 9 files
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Turn on auto-fix | Re-trigger cubic
… calls too Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would require human review. Fixes CLI/API mismatches for crawl --delay and scrape-family main-content defaults, but the default-output change is user-visible and --full-page doesn't suppress the interactive wizard; a human should decide before shipping.
View guided diff | Turn on auto-fix | Re-trigger cubic
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two CLI defaults disagreed with the API.
Crawl
--delayis sent in seconds.--delay <ms>is documented in milliseconds, but its value went to the API unchanged, and the API readsdelayas seconds (at most 60). So--delay 1000asked for 1000 seconds and was refused. The flag keeps its millisecond unit, and the CLI now divides by 1000 before sending:--delay 1500sendsdelay: 1.5.Scrapes follow the API's main-content default.
scrape,download,batch-scrapeandparsealways sentonlyMainContent: false, so their output kept navigation, headers and footers, unlike the API, the SDKs andsearch --scrape.onlyMainContentonly when a flag sets it, so by default the API returns the main content.--full-pageflag sendsonlyMainContent: falsefor the old output.--only-main-contentstill works.downloadwizard's "Only main content?" prompt now defaults to yes, and answering no sendsfalse.Visible change, for the next release notes.
firecrawl scrape,download,batch-scrapeandparsenow return the page's main content by default, without navigation, headers or footers. Add--full-pageto keep the whole page as before.crawl --delaynow works as documented, in milliseconds.This is meant to ship in the release after 1.26.24. The README option tables mention
--full-page.Tests. New tests cover:
--full-page, and both flags together;--full-page;The batch-scrape and crawl tests that asserted the old request bodies now assert the new ones. I also checked the built CLI against a local capture server: plain
scrapesends noonlyMainContent,--full-pagesendsfalse,--only-main-contentsendstrue, andcrawl --delay 1500sendsdelay: 1.5. Type check and build pass, and so does the full suite (877 tests).