Give it a list of business websites.
Get back clean text and verified facts, ready for descriptions and bullet points.

---

What it is

RapidSpider reads a list of business websites, fetches each one, and keeps two things:

1. The real text of the site, with menus, cookie banners and boilerplate removed.
2. Verified facts a writer needs for a long description, a short description and bullet points: services, opening hours, years in business, accreditations, prices and more.

One config.yml drives everything. An AI model called Laya runs inside the program and makes the judgement calls (is this page parked? is this line boilerplate? does this sentence state an opening time?). No API keys. No separate server.

You give it	You get back
A CSV, SQLite or JSON list with an id and a url (name, phone, address and email are optional)	One folder of SQLite files: cleaned site text, every page, every fact with its confidence and place on the page, one JSON profile per site, and a log of every decision

---

Start in 5 minutes

1. Copy the repo

git clone https://github.com/QudsLab/RapidSpider.git
cd RapidSpider

2. Install

pip install torch                  # CPU build. With an NVIDIA GPU install the CUDA build from pytorch.org instead
pip install -r requirements.txt
playwright install chromium        # one time

3. Add your sites

Create input/sites.csv. This folder is ignored by git, so your list stays yours.

id,website,name,town
1,https://example-practice.co.uk,Example Practice,Leeds
2,https://another-practice.co.uk,Another Practice,Bristol

Only id and website are required. Name and town help RapidSpider prove that a social or directory page is really about the business.

4. Run

python main.py config.yml

The first run downloads the Laya models once (a few GB). After that it works from the local cache.

5. Watch and read

Open http://127.0.0.1:8800 to watch it live. Results are in output/dental/.

Stopping: press Ctrl+C. Every browser RapidSpider started closes with it. Sites that were not finished stay pending and continue next time.

More ways to run it

python main.py config.yml --only 12,34           # only these input ids, good for testing one site
python main.py config.yml --requeue no_content   # run finished sites with that status again

Requirements

	Needed	Notes
Python	3.10 or newer	Developed on 3.14
Packages	pip install -r requirements.txt	Includes laya, which needs PyTorch
Browser	playwright install chromium	Used from the second step of the fetch ladder
Models	Downloaded on the first run	Internet needed once, then works from cache
GPU	Optional	With CUDA all three Laya models load. On CPU only English loads and runs are slower
RAM	About 10 GB free	laya.max_ram_mb unloads the least-used model above the limit
Disk	A few GB for models, plus the output
Tor	Only for the browser_tor step	Run Tor on 127.0.0.1:9050 and set network.tor_socks. Remove the step from fetch.ladder if you do not use it

---

Proof it works

This is the built-in dashboard after a real run on a business directory: nothing running, queue finished.

5,978 sites in that run	11,203 sites in the output	68,342 facts stored	20,846 pages fetched	3,336 site profiles
1,296,317 Laya questions in that run	2,273,775 Laya questions logged	10 ms per question (GPU)	3 models english, multilingual, typed-decisions	0 errors reported by Laya

Every number above is read from the screenshot. Sites that fail (error, blocked) are expected on the open web: some refuse every request. They are stored with the reason, and error sites are retried on the next run.

---

How it works

flowchart LR
    A[Your list] --> B{Fetch ladder}
    B -->|http| C[Classify the page]
    B -->|browser| C
    B -->|headed browser| C
    B -->|browser + Tor| C
    C -->|real site| D[Crawl a few pages]
    C -->|holding or Cloudflare| W[Wait, then retry]
    W --> C
    C -->|parked or dead| X[Stop: dead]
    C -->|third party| T[Keep one page only if it proves it is this business]
    D --> E[Clean the text]
    T --> E
    E --> F[Facts: Laya picks, then re-checks]
    F --> G[(SQLite output)]

Loading

1. Fetch. The home page is requested with plain HTTP first. If it is not accepted, RapidSpider escalates: headless browser, visible browser, then browser through Tor.
2. Classify. Each page is recognised as normal, a "please wait" shield, a Cloudflare check, a dead or parked domain, a third-party page (social profile, directory, group site) or empty. classes: in the config says what to do for each.
3. Crawl. A few more pages come from the sitemap and links, ranked by Laya.
4. Clean. Boilerplate, cookie text and near-duplicates are removed.
5. Facts. Laya finds the sentences that state each fact, then re-asks itself several times before it accepts one.
6. Save. Everything goes to the output folder.

Every site ends with one status:

Status	Meaning	Counted as
ok	Real site, text collected	finished with data
third_party	A social or directory page that proved it is this business	finished with data
dead	Domain gone or parked, confirmed more than once	finished, nothing to take
blocked	Login wall, or blocked on every step	finished, nothing to take
no_content	Page reached, but no real business text	finished, nothing to take
error, timeout, laya_down	Something failed	retried on the next run

---

Your input

CSV, SQLite or JSON. Map your own column names in config.yml:

input:
  path: input/sites.csv
  columns:
    id: id
    url: website
    name: name
    address: [town]          # a list is joined with ", "

---

Configuration

Everything lives in config.yml. Anything you leave out falls back to a built-in default, so a small file is enough.

Section	Controls
input	Where the site list is and which columns mean what
output	Output folder, per-day folders, database size limit
run	Threads, retries, time budget per site, resume
crawl	How many pages, sitemap use, text size limits
fetch	The ladder (http, browser, browser_headed, browser_tor) and timeouts
network	Tor address and how dead domains are confirmed
laya	CPU or GPU, models, memory limit, example sets
classes	What to do for each kind of page
parse	How aggressively boilerplate is dropped
facts	Which facts to collect and how strictly to check them
ui	The web interface address and port
logos	Optional logo collection

---

Facts

info.yml lists the ingredients a description needs. Each has a key, a label, a bullet-point shape and a plain-English description that Laya reads.

- key: opening_hours
  label: Opening hours
  bullet: "Open <days and hours>"
  description: "opening days and times, evening or Saturday appointments"
  patterns:
    - {re: '(?i)\b(?:mon(?:day)?)\s*(?:-|to)\s*(?:fri(?:day)?)[^.\n]{0,60}', value: 0}

How a fact is found and trusted:

1. Code first. Emails, phones, the business name and anything with a patterns regex are found by code, with no AI.
2. Laya reads the page in groups of eight numbered sentences and answers: which sentence states this ingredient, or none?
3. The sentence is the fact. Its place (top, middle, bottom, sentence number) is stored with it.
4. Laya double-checks. Each picked sentence is re-asked several times, worded differently with the answer keys flipped. It is accepted only when the rounds agree.

There is no manual review step.

---

Output

One folder per config, output/<name>/. Every database file rolls over to the next number before it reaches output.max_db_mb (45), so no file reaches 50 MB.

File	Contents
track.db	One row per site: status, reason, done code, attempts
sites_*.db	The cleaned text of each site and its contacts
pages_*.db	Each fetched page, with its full unfiltered text in raw
facts_*.db	Every fact: ingredient, sentence, confidence, votes, location, source page
profiles_*.db	One JSON profile per site. ready = 1 when enough categories are covered
laya_calls_*.db	Every question Laya was asked and its answer
incidents_*.db, run_log_*.db	Every ladder step and the run log
site_logs_*.db	One readable log per finished site
logos_*.db, logos/<id>/	Logo candidates and saved image files

-- how did the run go?
SELECT status, COUNT(*) FROM track GROUP BY status ORDER BY 2 DESC;

---

Web interface

While a run is going, open http://127.0.0.1:8800 (set under ui: in config.yml).

• Dashboard: progress, speed, outcomes, what is running now, pause, resume, thread count, stop.
• Tables: every output table has its own page. Click a column to sort, use the search box and drop-downs to filter, click a row for its detail.
• Laya questions: open any question to see the text it judged, its answer, the nearest solved example and the other checks on the same text.

---

Teaching Laya

Laya is shown solved examples next to every question, so you steer it by adding examples:

Where	Written by	Use
laya.examples in config.yml	you	Examples for your own niche
learned_examples.yml	you, in the web interface or by hand	Fixes to wrong answers
learned_auto.yml	RapidSpider	Decisions every verification round agreed on. Safe to delete, it is rebuilt

---

Documentation

Guide	What is in it
Getting started	Install, first run, CPU vs GPU
Configuration	Every setting, with defaults
The pipeline	Fetch ladder, page classes, crawling, third-party pages
Facts	info.yml, ingredients, verification
Laya	Examples, learning, how to correct it
Output	Every table and column, with queries
Web interface	Dashboard, tables, controls
Troubleshooting	Common problems and fixes

---

Credits

• Made with Claude by Anthropic. Claude helped build the code, the documentation and the graphics in this repository.
• Well tested at RightBiz.

---

Responsible use

RapidSpider reads robots.txt only to find sitemaps and does not enforce its rules. Check the terms of the sites you collect from and the data protection rules that apply to what you store.

License

MIT, see LICENSE.