Repository navigation
feat(importer): wire crawler to the rate.crawl_jobs queue - #4
Merged
Merged
Conversation
- QueueDrivenShopeeCrawler: claims jobs via dequeue_crawl_job, heartbeats long scrapes, checkpoints per-section ingest progress, acks/fails with the per-claim token fence - all through the SECURITY DEFINER RPCs (direct DML on crawl_jobs stays revoked server-side) - ShopeeIngestionPipeline.ingestProduct accepts an optional section report callback (merchant/master/listing/price/reviews) so the worker checkpoints after each idempotent section (plan v7 section 4.1) - Scripts/enqueue-shopee.ts: CLI to enqueue product URLs (parses shop/item ids, opens a crawl run) - Scripts/run-queue-crawl.ts: full driver - cloned-Chrome CDP launch (home IP default, USE_4G=1 for mproxy), human-like store warm-up before each PDP, worker loop until queue drains, finish_crawl_run + browser teardown in finally Verified against the live prod schema (migrations propagated): enqueue -> dequeue(attempt 1, token) -> checkpoint ok -> ack ok; re-enqueue after done correctly creates a fresh pending job (terminal states never block recrawl). tsc clean.
JOY (JOY)
force-pushed
the
feat/queue-wiring
branch
2 times, most recently
from
October 7, 2026 09:10
1ac5e04 to
aa6a1e9
Compare
Codex round-1 review of the queue wiring: - BLOCKER: a rejected heartbeat or refused ack now aborts the job with LEASE_LOST - the worker stops touching the listing and never logs a false ACKED (the true owner owns the outcome); the in-flight heartbeat promise is awaited on teardown - BLOCKER: post-run cleanup awaits finish_crawl_run in try/catch (the PostgREST builder has no .catch); stopLocalForwarder was missing on the branch base because dev had unpushed Shopee commits - dev is now synced (c941957) and the rebased branch has it - SHOULDs: setup after start_crawl_run is inside try/finally; kill is scoped to crawl profile dirs (single-driver documented); idle semantics document backoff-aware waiting (30s x 4); enqueue CLI exits non-zero on run failure, enqueue failure or skipped URLs; --max-jobs NaN guard; reviews-section checkpoint fires on empty review lists; checkpoint trail documented as observability (retries re-run the idempotent pipeline)
JOY (JOY)
force-pushed
the
feat/queue-wiring
branch
from
October 7, 2026 09:11
aa6a1e9 to
e440e94
Compare
added 2 commits
October 7, 2026 16:17
…are catch, chained heartbeats - re-check leaseLost after every awaited checkpoint inside the ingest report callback: the pipeline must not start the next section's writes once the lease died mid-checkpoint - catch treats leaseLost=true as abandonment even when the work threw a normal error afterwards - fail_crawl_job must never run for a job this worker no longer owns - heartbeats chain onto the in-flight promise so overlapping interval ticks cannot overwrite the tracked heartbeat (teardown awaits them all)
Codex round-3: chaining queued an unbounded heartbeat backlog when RPCs were slow and teardown awaited all of it. A tick now no-ops while the previous heartbeat is in flight (lease >> interval, so a skipped tick only shortens the effective lease marginally); teardown awaits just the one running RPC; hbInFlight resets to null on completion so the cast workaround is gone.
JOY (JOY)
force-pushed
the
feat/queue-wiring
branch
from
October 7, 2026 09:22
17b3e04 to
3ce6f27
Compare
Qwen review: the check-then-insert pattern is not atomic across concurrent workers; the DB unique index arbitrates, so a 23505 (or 'duplicate key') on the reviews/price inserts is a lost dedupe race, not a failure - warn and continue. All other errors still fail the ingestion.
JOY (JOY)
force-pushed
the
feat/queue-wiring
branch
from
October 7, 2026 09:31
86bb28c to
f5d3300
Compare
…f swallowing the batch Codex P1: batch inserts are atomic - swallowing a 23505 dropped the genuinely-new rows sharing the batch while the job still acked. On a race the worker now refetches which ids exist, retries the remainder once, and throws if rows remain unexplained.
JOY (JOY)
force-pushed
the
feat/queue-wiring
branch
from
October 7, 2026 09:32
f5d3300 to
a10d722
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the Shopee crawler to the queue schema merged in DOS-Me PR #1027 (now live on prod - verified start/finish_crawl_run, crawl_jobs, review_flags, price_history.observation_id all present).
Verification