Skip to content

Commit 7504628

Browse files
VinciGit00claude
andcommitted
feat: migrate CLI to scrapegraph-js v2 API
Full migration of just-scrape to the v2 API via scrapegraph-js PR #13 (pinned to head 096c110). The CLI surface, commands, and flags all move to v2 semantics and a single src/commands.ts holds every endpoint. ## CLI surface - extract, search, scrape, markdownify, crawl, monitor, history, credits - monitor gains full lifecycle: create, list, get, update, delete, pause, resume, and activity (paginated tick history with --limit / --cursor) - scrape supports 8 formats (markdown, html, screenshot, branding, links, images, summary, json) plus multi-format via comma-separated -f - crawl uses the v2 formats array and supports multi-format - search gains --country, --time-range, --format - Removed from API: agentic-scraper, generate-schema, sitemap, validate - --stealth is now a separate boolean; fetch mode is --mode auto|fast|js ## Environment - New default base URL: https://api.scrapegraphai.com/api/v2 - SGAI_TIMEOUT (was SGAI_TIMEOUT_S); legacy SGAI_TIMEOUT_S and JUST_SCRAPE_TIMEOUT_S are bridged transparently - SGAI_API_KEY, SGAI_API_URL, SGAI_DEBUG honored; JUST_SCRAPE_* still bridged ## Refactor - 8 per-endpoint files under src/commands/ merged into one src/commands.ts with named exports; cli.ts imports them statically - tsup output: single dist/cli.mjs (~30 KB) instead of 8 chunks - biome override scoped to src/commands.ts - scrapegraph-js ships without a prebuilt dist/, so CI builds it in-place after install Version bumped 0.3.0 → 1.0.0 to track SDK v2.0.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 2dc4981 commit 7504628

24 files changed

Lines changed: 1166 additions & 902 deletions

.github/workflows/ci.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@ jobs:
1414
- uses: actions/checkout@v4
1515
- uses: oven-sh/setup-bun@v2
1616
- run: bun install
17+
# scrapegraph-js is pinned to a GitHub commit (not yet on npm) and ships
18+
# without a prebuilt dist/ — build it in-place so module resolution works.
19+
- run: cd node_modules/scrapegraph-js && bun install && bun run build
1720
- run: bun test
1821

1922
lint:
@@ -23,4 +26,5 @@ jobs:
2326
- uses: actions/checkout@v4
2427
- uses: oven-sh/setup-bun@v2
2528
- run: bun install
29+
- run: cd node_modules/scrapegraph-js && bun install && bun run build
2630
- run: bun run check

README.md

Lines changed: 155 additions & 168 deletions
Large diffs are not rendered by default.

biome.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
},
1717
"overrides": [
1818
{
19-
"include": ["tests/**"],
19+
"include": ["tests/**", "src/commands.ts"],
2020
"linter": {
2121
"rules": {
2222
"suspicious": {

bun.lock

Lines changed: 4 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "just-scrape",
3-
"version": "0.2.1",
3+
"version": "1.0.0",
44
"description": "ScrapeGraph AI CLI tool",
55
"type": "module",
66
"main": "dist/cli.mjs",
@@ -28,7 +28,7 @@
2828
"chalk": "^5.4.1",
2929
"citty": "^0.1.6",
3030
"dotenv": "^17.2.4",
31-
"scrapegraph-js": "^1.0.0"
31+
"scrapegraph-js": "github:ScrapeGraphAI/scrapegraph-js#096c110"
3232
},
3333
"devDependencies": {
3434
"@biomejs/biome": "^1.9.4",

0 commit comments

Comments
 (0)