diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 079ad59..8c6a638 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,24 @@ +## 2026-10-09 — Gremlin and the mask hop chain agree (interop/tinkerpop, A1) + +`interop/tinkerpop/` adds a parity harness, built outside `consumers/` so the +`java-suites` CI job (which compiles `consumers/` without TinkerPop) does not +see TinkerPop. Two extra steps at the end of that job fetch the pinned jars and +run the harness. It copies three `RowStore.openWithEdges` fixtures into TinkerGraph +3.7.7, a test-oracle copy under rule E2. It then compares +`Graph.from(seeds).hop(KNOWS)^k.count()` with +`g.V(seeds).out("knows")^k.dedup().count()` for k = 1..3. + +- **Measured (JDK 28-ea+18, abi 0.12, avx512):** 25 checks green. All 9 + counts are equal, the pinned fixture reproduces GraphHopTest's 19 and 29, + and the largest case is n=20000, k=3, with 873 distinct rows. +- **Walks are not vertices:** the counts without `dedup()` are higher on every + fixture (e.g. 76 vs 55). Only the `dedup()` form may be lowered to a mask + chain, and the suite pins that it can tell the difference. +- **Disable-verified:** with the chain one hop short, all 9 comparisons go red. +- Dependencies: 19 jars pinned with sha256 in `deps.lock`. `fetch-deps.sh` + retries on HTTP 429 from Maven Central. +- **Not done:** a provider strategy (A2). Nothing in `src/main` changed. + ## 2026-09-22 — the Java gate is DISPATCHED, and one measurement stops being restated in eight places `ISS-LGJ-CONSUMERS-HAVE-NO-CI-LINE` is RESOLVED. `java-suites` in diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index 668fcb3..0e2bb87 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -8,6 +8,25 @@ > anti-pattern the imported board rules name. Backfilled below in one > pass rather than left stale; PR #4 onward gets its entry at merge time. +## PR (open) — A1 TinkerPop parity harness (branch `ccr-70e679fa-9qvl9y`) + +**Entry written at open.** + +- **Added:** `interop/tinkerpop/`, consisting of `TinkerParityTest`, + `deps.lock` (19 jars, sha256), `fetch-deps.sh`, `run.sh`, a README, and + gitignore lines for `lib/` and `out/`. +- **Locked:** the lance-graph-java hop chain equals the Gremlin `dedup()` + count for k = 1..3 on three fixtures. The walk count without `dedup()` + differs, and the suite requires that difference. +- **Deferred:** A2 (a `TraversalStrategy` that lowers compilable step runs + with a counted fallback). +- **CI:** two steps at the end of `java-suites` fetch the sha256-pinned jars + and run the harness on the job's JDK 28 EA. +- **Gates:** 25 checks green; disable run red-then-green. The core suite was + untouched (612 checks green before the change). +- **Confidence:** high for parity on these fixtures. It says nothing yet about + the speed or allocation of any lowered path (A3). + ## PR #81 — Panama × Valhalla as ONE production membrane (merged `07aa441`, branch `claude/great-pascal-k96kok`, head `1c66091`, 3 commits) **Entry written at merge**, not backfilled. diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 8d1d088..40306a6 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -343,3 +343,17 @@ jobs: exit 1 fi echo "all four consumer suites passed" + # Quick win A1: TinkerGraph 3.7.7 vs mask hop chain parity. Kept out of + # `consumers/` because that tree is compiled above without TinkerPop on the + # classpath. `fetch-deps.sh` downloads 19 jars from Maven Central and + # refuses any whose sha256 differs from `deps.lock`; `run.sh` reads + # LGJ_JDK, not JAVA_HOME, so point it at the JDK 28 EA that setup-java + # installed in this job. + - name: Fetch TinkerPop jars (sha256-pinned) + working-directory: lance-graph-java/interop/tinkerpop + run: ./fetch-deps.sh + - name: Run the TinkerPop parity harness (A1) + working-directory: lance-graph-java/interop/tinkerpop + env: + LGJ_JDK: ${{ env.JAVA_HOME }} + run: ./run.sh diff --git a/.gitignore b/.gitignore index 317c5ea..e0a30fd 100644 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,7 @@ Cargo.lock.bak valhalla-lab/reproducers/*.so valhalla-lab/reproducers/r8_standalone valhalla-lab/reproducers/r8-ndarray/target/ + +# interop/tinkerpop: fetched jars and compiled classes (fetch-deps.sh / run.sh) +/interop/tinkerpop/lib/ +/interop/tinkerpop/out/ diff --git a/interop/tinkerpop/README.md b/interop/tinkerpop/README.md new file mode 100644 index 0000000..c5bd3f1 --- /dev/null +++ b/interop/tinkerpop/README.md @@ -0,0 +1,33 @@ +# interop/tinkerpop: A1 parity harness + +One seeded graph, two engines, the same answers. The harness opens a +`RowStore.openWithEdges` fixture and copies its edges into **TinkerGraph** 3.7.7, +the TinkerPop line JanusGraph 1.x builds on. It then answers k-hop expansions +from the same seeds twice: + +| Engine | Query | +|---|---| +| lance-graph-java | `Graph.from(seeds).hop(KNOWS)…count()`, a mask hop chain | +| Gremlin | `g.V(seeds).out("knows")….dedup().count()` | + +```sh +./fetch-deps.sh # once: 19 jars from Maven Central, each sha256-checked against deps.lock +./run.sh # needs JDK 28 (LGJ_JDK, default /opt/jdks/jdk-28) and native/lgj-abi built --release +``` + +The harness lives outside `consumers/` on purpose. CI compiles every file under +`consumers/` against `java/out` alone, and TinkerPop is not on that classpath. +The `java-suites` job runs this harness in two separate steps at its end: +`fetch-deps.sh`, then `run.sh` with `LGJ_JDK` set to the job's JDK 28 EA. + +## What it checks (25 checks, all green) + +- The copy has edges, so a match is not between two empty sets. +- k = 1, 2, 3: the hop-chain count equals the Gremlin `dedup()` count on every fixture. The pinned fixture reproduces GraphHopTest's own 19 and 29. +- Falsifier: hopping over a classid with no edges does not match Gremlin. A second falsifier, run by hand: with the chain one hop short, every comparison goes red. +- Walks versus vertices: `out()…count()` without `dedup()` counts walks, which come out higher (for example 76 against 55). The suite requires at least one fixture where they differ, proving it can tell a bag from a set. Only the `dedup()` form may ever be lowered to a mask chain. + +## Rules it follows + +- The TinkerGraph copy reads facets one row at a time (`classidAt`, `payloadHi32At`, `payloadLow64At`). That is a scalar test oracle, which lance-graph-java licenses only in test code (rule E2). It is never an execution path. +- Nothing here changes `src/main`. A provider strategy that rewrites Gremlin into mask chains is the next step (A2), and it must keep a counted, visible fallback for every traversal it does not lower. diff --git a/interop/tinkerpop/deps.lock b/interop/tinkerpop/deps.lock new file mode 100644 index 0000000..322de2c --- /dev/null +++ b/interop/tinkerpop/deps.lock @@ -0,0 +1,23 @@ +# TinkerGraph 3.7.7 runtime closure (the line JanusGraph 1.x builds on), resolved +# once with `mvn dependency:copy-dependencies` and pinned here. fetch-deps.sh +# downloads each jar from Maven Central and refuses any checksum mismatch. +# Format: groupId:artifactId:version sha256 +com.carrotsearch:hppc:0.7.1 40d2a57f59e9eae7b018d3b4841954087ee40a5c1db6a54c3ea87742e3890391 +com.squareup:javapoet:1.13.0 4c7517e848a71b36d069d12bb3bf46a70fd4cda3105d822b0ed2e19c00b69291 +commons-beanutils:commons-beanutils:1.11.0 9e44ba68ec9a3f21286fa2a8bbb003b735c0f69101bb43144b79f4f8aaa74709 +commons-collections:commons-collections:3.2.2 eeeae917917144a68a741d4c0dff66aa5c5c5fd85593ff217bced3fc8ca783b8 +commons-io:commons-io:2.22.0 2b9a7b1f726fb86216dbd2c8321eabe0221dbd5b1be81c18e1cb53811b104758 +net.objecthunter:exp4j:0.4.8 271f7824ee8a3468257bc0613afdabb67597af8389317643fa806b983b7ecb27 +org.antlr:antlr4-runtime:4.9.1 a80502c3140ae7acfbb1f57847c8eb1c101461a969215ec38c3f1ebdff61ac29 +org.apache.commons:commons-configuration2:2.15.1 61ba5eaab8a17087042fe3d6811573bb85ccbdd6bd9d21ad464e6e20559dd1e8 +org.apache.commons:commons-lang3:3.20.0 69e5c9fa35da7a51a5fd2099dfe56a2d8d32cf233e2f6d770e796146440263f4 +org.apache.commons:commons-text:1.15.0 58d2da30f058512a1e7f914e39241deca4dff5c27a085b4ed2faa9e7208067f6 +org.apache.tinkerpop:gremlin-core:3.7.7 71cb32443cbdd9f925083be676a436c5ee8d14c097e71bd9fd73fd0a92d47bc8 +org.apache.tinkerpop:gremlin-language:3.7.7 44a1a80c046bc9a2b7832751039861ab5e69fe582d7564fde5c016ee6cb3724b +org.apache.tinkerpop:gremlin-shaded:3.7.7 ed55f024b149cd34f8be54dd8b7d09df8ab56c3d1c1225d3ca9b2af569d392a1 +org.apache.tinkerpop:tinkergraph-gremlin:3.7.7 05648d37f7e1c11b16440033307b984aa1541fd75a26c15c95d8d7203c77ebcc +org.javatuples:javatuples:1.2 2eda5b19d9820e1cc2f69fcd01639a715a673c11f8507e3d1ed593cf765d5e0a +org.slf4j:jcl-over-slf4j:1.7.25 5e938457e79efcbfb3ab64bc29c43ec6c3b95fffcda3c155f4a86cc320c11e14 +org.slf4j:slf4j-api:1.7.36 d3ef575e3e4979678dc01bf1dcce51021493b4d11fb7f1be8ad982877c16a1c0 +org.slf4j:slf4j-nop:1.7.36 c214958b07816cb4412b30c7bdbd4308ffdc6ba2a83767b8f3a9229cbd9274d6 +org.yaml:snakeyaml:2.0 880c9d896e4b74a06c549c15ca496450165d6909fa15d7e662bee8f6a66d7afa diff --git a/interop/tinkerpop/fetch-deps.sh b/interop/tinkerpop/fetch-deps.sh new file mode 100755 index 0000000..80348dd --- /dev/null +++ b/interop/tinkerpop/fetch-deps.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Download the pinned TinkerGraph closure into ./lib, verifying every checksum. +set -euo pipefail +cd "$(dirname "$0")" +mkdir -p lib +repo=https://repo1.maven.org/maven2 +grep -v '^#' deps.lock | while read -r coord sum; do + [ -z "$coord" ] && continue + IFS=: read -r g a v <<<"$coord" + jar="lib/$a-$v.jar" + if [ ! -f "$jar" ]; then + url="$repo/${g//./\/}/$a/$v/$a-$v.jar" + # Maven Central answers 429 under bursts; back off and retry a few times. + for delay in 0 2 4 8 16; do + sleep "$delay" + # Bound each attempt: a stalled transfer must time out so the loop advances. + if curl -fsS --connect-timeout 20 --max-time 120 -o "$jar.part" "$url"; then break; fi + done + mv "$jar.part" "$jar" + fi + echo "$sum $jar" | sha256sum -c --quiet - || { echo "checksum mismatch: $jar" >&2; rm -f "$jar"; exit 1; } +done +echo "deps ok: $(grep -cv -e '^#' -e '^$' deps.lock) jars" diff --git a/interop/tinkerpop/run.sh b/interop/tinkerpop/run.sh new file mode 100755 index 0000000..60d415e --- /dev/null +++ b/interop/tinkerpop/run.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Compile lance-graph-java, the graph consumer and this harness, then run it. +# Needs: JDK 28 (LGJ_JDK, default /opt/jdks/jdk-28; JAVA_HOME is not used, +# because it often points at an older JDK), the native library built in +# native/lgj-abi (cargo build --release), and ./fetch-deps.sh run once. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +root="$(cd "$here/../.." && pwd)" +J="${LGJ_JDK:-/opt/jdks/jdk-28}/bin" +SO="$root/native/lgj-abi/target/release/liblgj_abi.so" +out="$here/out" +rm -rf "$out" && mkdir -p "$out" +# The classpath is exactly the jars pinned in deps.lock, never a lib/ wildcard: +# a stale jar left in lib/ must not shadow a pinned one. +cp="" +while read -r coord _; do + [ -z "$coord" ] && continue + IFS=: read -r _ a v <<<"$coord" + jar="$here/lib/$a-$v.jar" + [ -f "$jar" ] || { echo "missing $jar; run ./fetch-deps.sh" >&2; exit 1; } + cp="$cp$jar:" +done < <(grep -v '^#' "$here/deps.lock") +if ! "$J/javac" --release 28 --enable-preview -d "$out" -cp "$cp" \ + $(find "$root/java/src/main" "$root/java/src/test" "$root/consumers/graph/src/main" "$here/src" -name '*.java') \ + >"$out/javac.log" 2>&1; then + cat "$out/javac.log" >&2 + echo "compile failed" >&2 + exit 1 +fi +exec "$J/java" --enable-preview --enable-native-access=ALL-UNNAMED -Dlgj.library="$SO" \ + -cp "$out:$cp" com.adaworldapi.interop.tinkerpop.TinkerParityTest diff --git a/interop/tinkerpop/src/com/adaworldapi/interop/tinkerpop/TinkerParityTest.java b/interop/tinkerpop/src/com/adaworldapi/interop/tinkerpop/TinkerParityTest.java new file mode 100644 index 0000000..b3fcdaf --- /dev/null +++ b/interop/tinkerpop/src/com/adaworldapi/interop/tinkerpop/TinkerParityTest.java @@ -0,0 +1,181 @@ +package com.adaworldapi.interop.tinkerpop; + +import com.adaworldapi.graph.Edge; +import com.adaworldapi.graph.Graph; +import com.adaworldapi.lancegraph.Checks; +import com.adaworldapi.lancegraph.FacetId; +import com.adaworldapi.lancegraph.NativeRuntime; +import com.adaworldapi.lancegraph.RowStore; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal; +import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; + +/** + * Quick win A1: one seeded graph, two engines, the same answers. + * + *

For each fixture, a {@link RowStore} is opened with {@link RowStore#openWithEdges}, and the + * same edges are copied into a TinkerGraph (TinkerPop's reference in-memory implementation). Then + * k-hop expansions from the same seed rows are answered twice: + * + *

+ * + *

The copy into TinkerGraph is test code only. It reads every row's facets one + * at a time ({@code classidAt} / {@code payloadHi32At} / {@code payloadLow64At}), the same scalar + * decoding {@code GraphHopTest}'s reference oracles use. lance-graph-java licenses a per-row Java + * loop only as a test oracle (rule E2), never as an execution path. + * + *

Walks are not vertices. {@code out().out().count()} counts walks, duplicates + * included; a mask hop chain counts distinct rows. Only the {@code dedup()} form is comparable. The + * suite records both, and requires at least one fixture where they differ, so the harness is shown + * able to tell them apart. + */ +public final class TinkerParityTest { + + private TinkerParityTest() {} + + private static final int FACETS_PER_ROW = 32; + private static final String KNOWS = "knows"; + private static final int MAX_HOPS = 3; + + private record Fixture(String name, long nRows, long seed, long gateMask, int radius, + long[] seeds) {} + + private static long[] stride(int count, long step, long offset) { + long[] rows = new long[count]; + for (int i = 0; i < count; i++) { + rows[i] = i * step + offset; + } + return rows; + } + + private static List fixtures() { + return List.of( + // GraphHopTest's pinned fixture: 1 hop reaches 19 rows, 2 hops 29. + new Fixture("pinned n=2000", 2000L, 0xF00D_CAFEL, 0x0L, 25, stride(10, 37, 5)), + new Fixture("n=10000 r=60", 10_000L, 0x5EED_1234L, 0x0L, 60, stride(40, 241, 3)), + new Fixture("n=20000 r=120", 20_000L, 0xBADC_0FFEL, 0x0L, 120, + stride(100, 197, 11))); + } + + public static void main(String[] args) { + System.out.println("TinkerParityTest"); + if (!NativeRuntime.isAvailable()) { + System.exit(Checks.reportUnavailable("TinkerParityTest")); + } + Checks c = new Checks("TinkerParityTest"); + boolean[] walksDifferedSomewhere = {false}; + for (Fixture f : fixtures()) { + runFixture(c, f, walksDifferedSomewhere); + } + c.section("walk count vs distinct count"); + c.that("on at least one fixture, out()...count() (walks) differs from the dedup count," + + " so the harness can tell bag from set", walksDifferedSomewhere[0]); + System.exit(c.report()); + } + + private static void runFixture(Checks c, Fixture f, boolean[] walksDiffered) { + c.section(f.name()); + try (RowStore store = RowStore.openWithEdges(f.nRows(), f.seed(), Edge.KNOWS, + f.gateMask(), f.radius())) { + TinkerGraph tg = copyIntoTinkerGraph(store); + long edges = tg.traversal().E().count().next(); + c.that("TinkerGraph copy has edges (" + edges + "), so the comparison is not vacuous", + edges > 0); + GraphTraversalSource g = tg.traversal(); + Object[] seedIds = boxed(f.seeds()); + + for (int k = 1; k <= MAX_HOPS; k++) { + long lance = lanceHopCount(store, f.seeds(), Edge.KNOWS, k); + long gremlinSet = outN(g.V(seedIds), k).dedup().count().next(); + long gremlinWalks = outN(g.V(seedIds), k).count().next(); + c.note(String.format("k=%d lance-graph=%d gremlin dedup=%d gremlin walks=%d", + k, lance, gremlinSet, gremlinWalks)); + c.eq("k=" + k + ": lance-graph hop chain equals Gremlin dedup count", + gremlinSet, lance); + c.that("k=" + k + ": walks >= distinct vertices", gremlinWalks >= gremlinSet); + if (gremlinWalks != gremlinSet) { + walksDiffered[0] = true; + } + if (k == 1 && gremlinSet > 0) { + // Falsifier: the comparison must be able to fail. Hopping a classid that + // has no edges must not reproduce Gremlin's answer. + long wrong = lanceHopCount(store, f.seeds(), Edge.KNOWS + 1, k); + c.notEq("k=1: a hop over the wrong edge classid does NOT match Gremlin", + gremlinSet, wrong); + } + } + tg.close(); + } catch (Exception e) { + throw new IllegalStateException(e); + } + } + + private static GraphTraversal outN(GraphTraversal t, int k) { + for (int i = 0; i < k; i++) { + t = t.out(KNOWS); + } + return t; + } + + private static long lanceHopCount(RowStore store, long[] seeds, int edgeClassid, int hops) { + List chain = new ArrayList<>(); + try { + Graph g = Graph.open(store); + chain.add(g); + g = g.from(seeds); + chain.add(g); + for (int i = 0; i < hops; i++) { + g = g.hop(edgeClassid); + chain.add(g); + } + return g.count(); + } finally { + for (Graph g : chain) { + g.close(); + } + } + } + + /** Test-oracle copy: every KNOWS facet becomes one Gremlin edge, duplicates kept. */ + private static TinkerGraph copyIntoTinkerGraph(RowStore store) { + TinkerGraph tg = TinkerGraph.open(); + long n = store.rowCount(); + Vertex[] vs = new Vertex[(int) n]; + for (long row = 0; row < n; row++) { + vs[(int) row] = tg.addVertex(T.id, row); + } + for (long row = 0; row < n; row++) { + for (int f = 0; f < FACETS_PER_ROW; f++) { + FacetId facet = FacetId.of(f); + if (store.classidAt(row, facet) != Edge.KNOWS) { + continue; + } + if (store.payloadHi32At(row, facet) != 0) { + continue; + } + long target = store.payloadLow64At(row, facet); + if (target >= 0 && target < n) { + vs[(int) row].addEdge(KNOWS, vs[(int) target]); + } + } + } + return tg; + } + + private static Object[] boxed(long[] rows) { + Object[] out = new Object[rows.length]; + for (int i = 0; i < rows.length; i++) { + out[i] = rows[i]; + } + return out; + } +}