Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
55a7f42
feat(v2): add zod schemas and types for v2 API
FrancescoSaverioZuppichini Apr 14, 2026
97a8958
feat(v2): implement SDK with crawl and monitor namespaces
FrancescoSaverioZuppichini Apr 14, 2026
5847b95
refactor: inline namespace functions directly in objects
FrancescoSaverioZuppichini Apr 14, 2026
a725c1c
fix: use deleted status and move types to src/types.ts
FrancescoSaverioZuppichini Apr 14, 2026
11b811c
feat(v2): rewrite tests for v2 API
FrancescoSaverioZuppichini Apr 14, 2026
8f44af9
test: add comprehensive tests for formats, fetchConfig, and document …
FrancescoSaverioZuppichini Apr 14, 2026
13578aa
refactor: rename getHistory/getHistoryEntry to history.list/history.get
FrancescoSaverioZuppichini Apr 14, 2026
6844d33
chore: split unit and integration tests for CI
FrancescoSaverioZuppichini Apr 14, 2026
d71e4d9
docs: update examples for v2 API
FrancescoSaverioZuppichini Apr 14, 2026
e30cc55
docs: rewrite README for v2 API
FrancescoSaverioZuppichini Apr 14, 2026
c9153e4
fix: address code smells in examples and README
FrancescoSaverioZuppichini Apr 14, 2026
22b936e
docs: use local banner from media folder
FrancescoSaverioZuppichini Apr 14, 2026
b5926d1
fix: remove hardcoded API key from integration tests
FrancescoSaverioZuppichini Apr 14, 2026
46bed0e
fix: add node types to tsconfig
FrancescoSaverioZuppichini Apr 14, 2026
276713b
chore: remove .claude and CLAUDE.md, add to gitignore
FrancescoSaverioZuppichini Apr 14, 2026
0738786
minor changes
FrancescoSaverioZuppichini Apr 14, 2026
3703f8c
docs: reorganize examples table by service, add SGAI_API_KEY env var
FrancescoSaverioZuppichini Apr 14, 2026
33ea3e6
feat: add ScrapeGraphAI client pattern, remove generateSchema
FrancescoSaverioZuppichini Apr 14, 2026
0522abc
test: update integration tests to use ScrapeGraphAI client
FrancescoSaverioZuppichini Apr 14, 2026
7c7d31c
chore: ignore scripts directory
FrancescoSaverioZuppichini Apr 14, 2026
d4bad81
chore: bump version to 2.0.0
FrancescoSaverioZuppichini Apr 14, 2026
b9037f5
docs: update CONTRIBUTING.md for JS SDK
FrancescoSaverioZuppichini Apr 14, 2026
91f25ce
style: format package.json arrays
FrancescoSaverioZuppichini Apr 15, 2026
93a1b15
feat(examples): add polling loop to crawl examples
FrancescoSaverioZuppichini Apr 15, 2026
bebf149
chore: update CLAUDE.md, fix health endpoint, add playground script
FrancescoSaverioZuppichini Apr 15, 2026
2eba148
fix: update API base URL and rename timeout env var
FrancescoSaverioZuppichini Apr 15, 2026
096c110
feat: add monitor.activity endpoint and update examples
FrancescoSaverioZuppichini Apr 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .claude/commands/gh-pr.md

This file was deleted.

187 changes: 0 additions & 187 deletions .claude/skills/gh-pr.md

This file was deleted.

3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
scripts/
node_modules
dist/
.DS_Store
bun.lock
*.tsbuildinfo
.env
doc/
.DS_Store
.claude/
143 changes: 143 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Before completing any task

Always run these commands before committing or saying a task is done:

```bash
bun run format
bun run lint
bunx tsc --noEmit
bun run build
bun test
```

No exceptions.

## Project Overview

**scrapegraph-js** is the official JavaScript/TypeScript SDK for the ScrapeGraph AI API. It provides a TypeScript client for intelligent web scraping powered by AI.

## Repository Structure

```
scrapegraph-js/
├── src/ # TypeScript SDK source
├── tests/ # Test suite
├── examples/ # Usage examples
├── scripts/ # Development utilities
└── .github/workflows/ # CI/CD
```

## Tech Stack

- **Language**: TypeScript (Node.js 22+)
- **Runtime**: Bun
- **Core Dependencies**: zod (validation)
- **Testing**: Bun test
- **Code Quality**: Biome (lint + format)
- **Build**: tsup

## Commands

```bash
# Install
bun install

# Dev (watch mode)
bun run dev

# Test
bun test # unit tests
bun run test:integration # integration tests

# Format
bun run format

# Lint
bun run lint

# Type check
bunx tsc --noEmit

# Build
bun run build

# Playground (loads .env)
bun run playground
```

## Architecture

**Core Components:**

1. **Client** (`src/scrapegraphai.ts`):
- `ScrapeGraphAI()` - Factory function returning namespaced client
- Handles all API communication

2. **Types** (`src/types.ts`):
- Request/response types for all endpoints
- Zod schema inference

3. **Schemas** (`src/schemas.ts`):
- Zod validation schemas

4. **Config** (`src/env.ts`):
- Environment variable handling

## API Methods

| Method | Purpose |
|--------|---------|
| `sgai.scrape()` | AI data extraction from URL |
| `sgai.extract()` | Extract from raw HTML/text |
| `sgai.search()` | Web search + extraction |
| `sgai.crawl.start()` | Start crawl job |
| `sgai.crawl.get()` | Get crawl status |
| `sgai.monitor.create()` | Create monitoring job |
| `sgai.monitor.get()` | Get monitor status |
| `sgai.monitor.update()` | Update monitor config |
| `sgai.monitor.delete()` | Delete monitor |
| `sgai.credits()` | Check API credits |
| `sgai.healthy()` | Health check |
| `sgai.history.list()` | List request history |
| `sgai.history.get()` | Get specific request |

## Adding New Endpoint

1. Add types in `src/types.ts`
2. Add Zod schema in `src/schemas.ts`
3. Add function in `src/scrapegraphai.ts`
4. Wire into `ScrapeGraphAI()` client object
5. Export types in `src/index.ts`
6. Add tests in `tests/`
7. Add example in `examples/`

## Environment Variables

- `SGAI_API_KEY` - API key for authentication
- `SGAI_DEBUG` - Enable debug logging (optional)

## Usage

```typescript
import { ScrapeGraphAI } from "scrapegraph-js";

const sgai = ScrapeGraphAI(); // reads SGAI_API_KEY from env

const res = await sgai.scrape({
url: "https://example.com",
prompt: "Extract the main heading",
});

if (res.status === "success") {
console.log(res.data?.result);
}
```

## Links

- [API Docs](https://docs.scrapegraphai.com)
- [npm](https://www.npmjs.com/package/scrapegraph-js)
Loading
Loading