You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) is Anthropic's agentic coding tool that runs in your terminal. Since it can execute shell commands, it works seamlessly with `just-scrape`.
-`just-scrape sitemap <url> --json` — get all URLs from a sitemap
68
+
69
+
Use --schema to enforce a JSON schema on the output.
70
+
Use --stealth for sites with anti-bot protection.
71
+
```
72
+
73
+
### Example prompts
74
+
75
+
```
76
+
> Scrape the pricing page at https://example.com/pricing and create a comparison table
77
+
78
+
> Search for "best practices for REST API pagination" and summarize the top results
79
+
80
+
> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md
81
+
```
82
+
83
+
### Non-interactive / CI usage
84
+
85
+
```bash
86
+
claude -p "Use just-scrape to scrape https://example.com/changelog \
87
+
and extract the latest 5 releases. Save as CHANGELOG_SUMMARY.md"
88
+
```
89
+
41
90
## Manual setup with Cursor
42
91
43
92
If you are using Cursor without the skills.sh integration, configure `just-scrape` via the [MCP Server](/services/mcp-server/cursor) for the best experience.
Copy file name to clipboardExpand all lines: services/cli/ai-agent-skill.mdx
+76-1Lines changed: 76 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: 'AI Agent Skill'
3
3
description: 'Give AI coding agents direct access to web scraping through just-scrape'
4
4
---
5
5
6
-
`just-scrape` can be installed as a **skill** for AI coding agents via [Vercel's skills.sh](https://skills.sh). This lets agents like Claude, Cursor, and others call ScrapeGraphAI commands directly during a coding session.
6
+
`just-scrape` can be installed as a **skill** for AI coding agents via [Vercel's skills.sh](https://skills.sh). This lets agents like Claude Code, Cursor, and others call ScrapeGraphAI commands directly during a coding session.
[Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) is Anthropic's agentic coding tool that runs directly in your terminal. Since Claude Code can execute shell commands, it works seamlessly with `just-scrape`.
44
+
45
+
### Setup
46
+
47
+
<Steps>
48
+
<Steptitle="Install just-scrape globally">
49
+
```bash
50
+
npm install -g just-scrape
51
+
```
52
+
</Step>
53
+
<Steptitle="Set your API key">
54
+
Add the key to your shell profile (`~/.zshrc`, `~/.bashrc`, etc.) so Claude Code inherits it automatically:
55
+
```bash
56
+
export SGAI_API_KEY="sgai-..."
57
+
```
58
+
</Step>
59
+
<Steptitle="Start Claude Code and scrape">
60
+
Launch Claude Code and ask it to scrape any website:
61
+
```
62
+
claude
63
+
> Scrape https://news.ycombinator.com and extract the top 10 stories with titles and URLs using just-scrape
64
+
```
65
+
</Step>
66
+
</Steps>
67
+
68
+
### Add just-scrape to CLAUDE.md
69
+
70
+
To make Claude Code aware of `just-scrape` in every session, add instructions to the `CLAUDE.md` file in your project root (or `~/.claude/CLAUDE.md` for global access):
71
+
72
+
```markdown CLAUDE.md
73
+
## Web Scraping
74
+
75
+
This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
76
+
The API key is set via the SGAI_API_KEY environment variable.
77
+
78
+
Available commands (always use --json flag):
79
+
-`just-scrape smart-scraper <url> -p <prompt> --json` — AI extraction from a URL
80
+
-`just-scrape search-scraper <prompt> --json` — search the web and extract data
81
+
-`just-scrape markdownify <url> --json` — convert a page to markdown
0 commit comments