DIPs testing#67
Draft
MoonBoi9001 wants to merge 111 commits into
Draft
Conversation
MoonBoi9001
force-pushed
the
samuel/dips-dev-environment
branch
3 times, most recently
from
cb0c65b to
39e989f
Compare
MoonBoi9001
changed the title
MoonBoi9001
force-pushed
the
samuel/dips-dev-environment
branch
from
39e989f to
28089bd
Compare
MoonBoi9001
force-pushed
the
samuel/dips-dev-environment
branch
from
7c68181 to
0345ff8
Compare
MoonBoi9001
force-pushed
the
samuel/dips-dev-environment
branch
2 times, most recently
from
962e3f7 to
972b4af
Compare
Mount local checkouts of contracts, indexer-rs, dipper, iisa, and the eligibility-oracle-node so the stack runs your branches end-to-end. Also adds a separate eligibility-oracle overlay and aligns the base compose file with the dev overrides. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add dev run-dips.sh entrypoints for indexer-agent and indexer-service so they consume mounted source. Refresh dipper, iisa, eligibility-oracle, tap-agent, and tap-escrow-manager run scripts to drive the end-to-end DIPs pipeline against horizon contracts. Drop the iisa local-scoring stub set (replaced by API-push design from the cronjob). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add scripts for sending indexing requests, deploying test subgraphs, generating extra indexers, monitoring the DIPs pipeline, checking subgraph sync, and snapshotting network status. These drive the end-to-end testing flow against the local stack. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add reusable .claude/skills entries for the recurring DIPs testing operations (fresh deploy, add indexers, deploy subgraphs, send indexing request, network status). Add BUGS.md to log every issue surfaced during end-to-end testing, CLAUDE.md for project-level guidance, and TESTING-STATUS.md to track progress. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch the env file to the testing-targeted config: enable the indexing-payments profile, point all *_SOURCE_ROOT at local checkouts, pin contracts to mb9/dips-local-testing-fixes, and add the x402 receiver wallet for the gateway. Refresh .gitignore alongside. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Base stack now brings up every service from pinned commits or images. Per-service dips-*.yaml overlays opt individual components into a source-mount mode that builds in-container from a local checkout. dips.yaml stays as the mount-everything preset. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MoonBoi9001
force-pushed
the
samuel/dips-dev-environment
branch
from
9dd0111 to
77db3a0
Compare
The DIPs source-mount overlays were a parallel build pipeline that duplicated the image-only path. They hard-coded Mac host paths in the env file and broke on any non-Mac clone. Drop them; rely on the pinned image versions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cronjob image is published to GHCR by the upstream subgraph-dips-indexer-selection workflow. Pull it via image: instead of cloning the private source and building locally, so fresh deployments work without GitHub auth in the build container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The eligibility-oracle-node repo is private. Cloning at build time fails on machines without GitHub auth. Each developer drops a local clone at the gitignored containers/oracles/eligibility- oracle-node/source/ path; the Dockerfile COPYs from there. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
graph-contracts no longer writes tap-contracts.json — TAP-related addresses (GraphTallyCollector, PaymentsEscrow) live in horizon.json now. wait_for_config blocked 300s waiting for the missing file; dipper read TAPVerifier from it. Both updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
indexer-agent crashes at SubgraphClient.create when the spec has tapSubgraph or indexingPaymentsSubgraph as empty objects (truthy in JS). Restore both endpoints. The TAP subgraph isn't deployed on this branch, but a stale URL still lets the agent start. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The agent's @semiotic-labs/tap-contracts-bindings library has no chainId 1337 baked in. Without a tap-contracts.json address book the binding library rejects Network.create and the management API never starts. Stub it from horizon.json: TAPVerifier <- GraphTallyCollector, Escrow <- PaymentsEscrow. Addresses aren't exercised on the DIPs path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Rust tap-agent and indexer-service crash on startup with "missing field query_url for default.subgraphs.escrow" — the schema hard-requires this section even though semiotic/tap subgraph isn't deployed on this branch. Stale URL satisfies the schema; queries against it fail gracefully and the DIPs flow doesn't use this path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extras now use the same build context, image versions, healthchecks, and run.sh as the primary indexer-agent and indexer-service. Drops the dockerfile_inline wrapper, the host source-mount volumes, and the run-dips.sh entrypoint override. Per-indexer identity and hostnames flow in via compose environment overrides. Also fixes the generator's ENV_FILE path from a non-existent .environment to .env, so the COMPOSE_FILE entry is actually written. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dips overlay was removed in commit 325ec70; the add-indexers skill still chained it into every docker compose invocation, which would fail on a fresh clone with "no such file or directory". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Old skill was a soft reset (containers + volumes only) plus several references that no longer hold on this branch — the dips compose overlay was deleted, tap-escrow-manager was renamed, the TAP subgraph isn't deployed any more, and run.sh is no longer volume- mounted. The new skill targets the lnet-test VM, wipes containers, volumes, networks, all images, and the clone itself, then re-clones from origin, repopulates eligibility-oracle-node/source via rsync from the Mac, runs build --pull, brings up the stack, and streams per-service health to the user. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The old skill ran every command (docker, curl, python) against the local docker daemon. That doesn't work on the Mac+VM setup where docker lives on lnet-test and the generator script lives on the Mac. Rewrite makes the split explicit: generator runs on Mac, the yaml and updated .env are scp'd to the VM, all docker / docker-pause / curl-localhost commands are SSH-wrapped. Drop stale claims about flock-serialized cargo builds (no Rust compile happens any more — extras use primary's thin-wrapper Dockerfile), the legacy TAP subgraph, the DOCKER_DEFAULT_PLATFORM= prefix (VM is amd64-native), and the orphan reference to /fresh-deploy's down -v handling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The contract-address helper chained an overlay file that was removed earlier on this branch, breaking the script with "no such file or directory" on a fresh clone. Let docker compose read the overlay list from .env. Also drop an unused import. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The scripts hit localhost-only endpoints and shell out to cast and the graph CLI, so they must run on the VM via SSH. Document the one-time install of foundry from a release tarball, plus Node 22 from NodeSource since the apt default is too old for graph CLI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The start-indexing-extra init container deposited 2 GRT of query-fee escrow per extra indexer. The gateway-side escrow manager already funds that same escrow for every indexer with an active allocation, so the deposit was redundant and is removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KJmMVG736T3xefovirWryK
The start-indexing-extra init container sent 1 ETH to each extra indexer for gas. Those accounts are anvil junk-mnemonic accounts the chain already pre-funds, so raising anvil to `--accounts 20` covers them and the transfer is dropped. Operators still get their ETH. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KJmMVG736T3xefovirWryK
The gateway image cloned edgeandnode/gateway and ran a debug cargo build on every cold deploy (~minutes). It now layers run.sh onto the prebuilt ghcr.io/edgeandnode/graph-gateway image that production runs (v27.6.0, a release build), cutting one Rust compile from the deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KJmMVG736T3xefovirWryK
Ported from the private CI mirror after three consecutive green runs against pinned images. Dipper gets its own funded wallet so its transactions stop colliding with deploys and tests, and two new tests drive a DIPs request through acceptance to on-chain payment. See BUGS.md.
Main added a dev workflow that compiles a local gateway inside the Dockerfile's builder stage, while this branch had switched the gateway to the prebuilt published image. The merged file keeps the builder stage for that dev workflow and the prebuilt image as the runtime.
BUG-002/003/005 described code that no longer exists on this branch (the old indexer-service DIPs config, the retired TAP subgraph, the removed extra-indexer escrow deposits), so they're gone. Every remaining bug gains a status line; only BUG-007 and BUG-008 stay open.
Its fix is the obvious implementation now in dipper's startup script, a regression would fail loudly and immediately, and unlike the kept entries it documents no non-obvious guard.
The rewards-issuer registration now explains at the call site why skipping it bricks all allocations, and the dipper pin notes its rollback floor. With every fixed bug's lesson living in the tree, the tracker holds just the two open items: burst scale and unpaid allocations.
The two open items become BUG-001 (burst scale) and BUG-002 (unpaid allocations); the header notes the old numbering series was retired so historical references stay decodable.
The stranded-allocations bug turned out to be fixed upstream in the pinned indexer-agent (a rule reaper with a subgraph staleness guard), and dipper #661 now paces offers at the source. One re-run of the 50-request burst refreshes the scale numbers and verifies both.
Dipper now paces offers to network capacity and serves interactive requests first (dipper #661 and #662, both merged), so a burst queues instead of expiring. Both open bugs now await the same proof: one 50-request burst re-run on a pinned image containing the fixes.
Offer pacing and queue priority landed in dipper, and the stranded-allocation reaper ships in the pinned indexer-agent, so both entries are pruned. A 50-request burst re-run will either confirm the clean slate or re-document what it finds with fresh numbers.
The first CI run on the paced dipper failed 3 tests. The trail led to the indexer-agent closing a just-recreated allocation and stamping a never rule that blocklisted the deployment for DIPs; a test-hygiene entry covers the missing state restore on abort.
A merge added a dev build stage at the top of the gateway Dockerfile, stranding the version variable inside that stage where the image-selection FROM line cannot see it. Every cold build then failed with an empty image tag; declaring the variable first restores bring-up.
The stack stopped deploying the legacy TAP subgraph when payments moved to the Horizon contracts, but the sync checker still listed it and so exited non-zero on every otherwise-healthy run.
The postgres image boots a temporary socket-only server while running init scripts, then restarts into the real one. The health probe passed against that temporary server, letting graph-node connect just before the restart dropped it; probing over TCP waits for the real one.
The client for dipper's admin RPC was never part of the stack definition, so a fresh machine lacked it until someone pulled it by hand with a tag that goes stale on version bumps. A pull-only compose entry now pins it to the server's version; deploys fetch it.
Agreements could only collect payment 1 hour after acceptance, so verifying a payout meant waiting out the clock. A 2-minute floor (and 7-day ceiling) makes a full accept-then-collect cycle watchable in a session; terms only affect new agreements.
The image lookup used a flag that only rebases relative paths and never finds the compose file itself, so the command came back empty and docker tried to pull a nonsense image. Pointing compose at the file directly makes the lookup work from any directory.
Agents schedule each payment collection 50% of the way into the agreement's allowed window, so the previous 2-minute-to-7-day window meant a 3.5-day wait for the first payout. A 60-to-240-second window puts collections 150 seconds apart, watchable live.
The payment collector contract insists the window between collections spans at least 600 seconds, so the previous 60-to-240 test window made every offer revert on-chain. 60 to 660 is the narrowest legal window; agents collect 50% in, paying out every 6 minutes.
Dipper never checks its configured collection window against the contract's minimum and never decodes the revert reason, so a bad config loops on failing transactions while indexers wait for offers that cannot land. Filed as Bug 3 for an upstream fix.
The tracker header asked for a burst re-run on a dipper pin with offer pacing to prove the old burst-scale bug stayed fixed. A 43-request burst on 2026-07-07 accepted all 129 agreements in about 7.5 minutes with zero expiries or rejections, so the request is done.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dev environment for end-to-end DIPs testing against Horizon contracts. Sharing as a draft.