Skip to content

Commit 3cbb5fd

Browse files
committed
feat: update claude code rule
1 parent 6ce7b73 commit 3cbb5fd

2 files changed

Lines changed: 125 additions & 1 deletion

File tree

knowledge-base/cli/ai-agent-skill.mdx

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,55 @@ just-scrape search-scraper "latest release notes for react-query" \
3838
--num-results 3 --json
3939
```
4040

41+
## Using with Claude Code
42+
43+
[Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) is Anthropic's agentic coding tool that runs in your terminal. Since it can execute shell commands, it works seamlessly with `just-scrape`.
44+
45+
### Setup
46+
47+
1. Install `just-scrape` globally: `npm install -g just-scrape`
48+
2. Set `SGAI_API_KEY` in your shell profile (`~/.zshrc`, `~/.bashrc`)
49+
3. Launch Claude Code and ask it to scrape anything
50+
51+
### Add just-scrape to CLAUDE.md
52+
53+
Add scraping instructions to your `CLAUDE.md` (project root or `~/.claude/CLAUDE.md` for global):
54+
55+
```markdown CLAUDE.md
56+
## Web Scraping
57+
58+
This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
59+
The API key is set via the SGAI_API_KEY environment variable.
60+
61+
Available commands (always use --json flag):
62+
- `just-scrape smart-scraper <url> -p <prompt> --json` — AI extraction from a URL
63+
- `just-scrape search-scraper <prompt> --json` — search the web and extract data
64+
- `just-scrape markdownify <url> --json` — convert a page to markdown
65+
- `just-scrape crawl <url> -p <prompt> --json` — crawl multiple pages
66+
- `just-scrape scrape <url> --json` — get raw HTML
67+
- `just-scrape sitemap <url> --json` — get all URLs from a sitemap
68+
69+
Use --schema to enforce a JSON schema on the output.
70+
Use --stealth for sites with anti-bot protection.
71+
```
72+
73+
### Example prompts
74+
75+
```
76+
> Scrape the pricing page at https://example.com/pricing and create a comparison table
77+
78+
> Search for "best practices for REST API pagination" and summarize the top results
79+
80+
> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md
81+
```
82+
83+
### Non-interactive / CI usage
84+
85+
```bash
86+
claude -p "Use just-scrape to scrape https://example.com/changelog \
87+
and extract the latest 5 releases. Save as CHANGELOG_SUMMARY.md"
88+
```
89+
4190
## Manual setup with Cursor
4291

4392
If you are using Cursor without the skills.sh integration, configure `just-scrape` via the [MCP Server](/services/mcp-server/cursor) for the best experience.

services/cli/ai-agent-skill.mdx

Lines changed: 76 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: 'AI Agent Skill'
33
description: 'Give AI coding agents direct access to web scraping through just-scrape'
44
---
55

6-
`just-scrape` can be installed as a **skill** for AI coding agents via [Vercel's skills.sh](https://skills.sh). This lets agents like Claude, Cursor, and others call ScrapeGraphAI commands directly during a coding session.
6+
`just-scrape` can be installed as a **skill** for AI coding agents via [Vercel's skills.sh](https://skills.sh). This lets agents like Claude Code, Cursor, and others call ScrapeGraphAI commands directly during a coding session.
77

88
## Install the skill
99

@@ -38,6 +38,81 @@ just-scrape search-scraper "latest release notes for react-query" \
3838
--num-results 3 --json
3939
```
4040

41+
## Using with Claude Code
42+
43+
[Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) is Anthropic's agentic coding tool that runs directly in your terminal. Since Claude Code can execute shell commands, it works seamlessly with `just-scrape`.
44+
45+
### Setup
46+
47+
<Steps>
48+
<Step title="Install just-scrape globally">
49+
```bash
50+
npm install -g just-scrape
51+
```
52+
</Step>
53+
<Step title="Set your API key">
54+
Add the key to your shell profile (`~/.zshrc`, `~/.bashrc`, etc.) so Claude Code inherits it automatically:
55+
```bash
56+
export SGAI_API_KEY="sgai-..."
57+
```
58+
</Step>
59+
<Step title="Start Claude Code and scrape">
60+
Launch Claude Code and ask it to scrape any website:
61+
```
62+
claude
63+
> Scrape https://news.ycombinator.com and extract the top 10 stories with titles and URLs using just-scrape
64+
```
65+
</Step>
66+
</Steps>
67+
68+
### Add just-scrape to CLAUDE.md
69+
70+
To make Claude Code aware of `just-scrape` in every session, add instructions to the `CLAUDE.md` file in your project root (or `~/.claude/CLAUDE.md` for global access):
71+
72+
```markdown CLAUDE.md
73+
## Web Scraping
74+
75+
This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
76+
The API key is set via the SGAI_API_KEY environment variable.
77+
78+
Available commands (always use --json flag):
79+
- `just-scrape smart-scraper <url> -p <prompt> --json` — AI extraction from a URL
80+
- `just-scrape search-scraper <prompt> --json` — search the web and extract data
81+
- `just-scrape markdownify <url> --json` — convert a page to markdown
82+
- `just-scrape crawl <url> -p <prompt> --json` — crawl multiple pages
83+
- `just-scrape scrape <url> --json` — get raw HTML
84+
- `just-scrape sitemap <url> --json` — get all URLs from a sitemap
85+
86+
Use --schema to enforce a JSON schema on the output.
87+
Use --stealth for sites with anti-bot protection.
88+
```
89+
90+
### Example prompts for Claude Code
91+
92+
Once configured, you can ask Claude Code things like:
93+
94+
```
95+
> Scrape the pricing page at https://example.com/pricing and create a comparison table in pricing.md
96+
97+
> Search for "best practices for REST API pagination" and summarize the top 5 results
98+
99+
> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md
100+
101+
> Crawl https://competitor.com/blog and extract all article titles, dates, and summaries into a JSON file
102+
```
103+
104+
Claude Code will run the appropriate `just-scrape` command, parse the JSON output, and use the data to complete your task.
105+
106+
### Claude Code + just-scrape in CI
107+
108+
You can also use Claude Code with `just-scrape` in non-interactive mode for automated workflows:
109+
110+
```bash
111+
claude -p "Use just-scrape to scrape https://example.com/changelog \
112+
and extract the latest 5 releases with version numbers, dates, and highlights. \
113+
Save the result as CHANGELOG_SUMMARY.md"
114+
```
115+
41116
## Tips
42117

43118
- Set `SGAI_API_KEY` in your shell profile so the skill picks it up automatically.

0 commit comments

Comments
 (0)