Add interop/tinkerpop: Gremlin vs mask hop chain parity harness (A1)

.claude/board/LATEST_STATE.md

@@ -1,3 +1,24 @@
+## 2026-10-09 — Gremlin and the mask hop chain agree (interop/tinkerpop, A1)
+
+`interop/tinkerpop/` adds a parity harness, built outside `consumers/` so the
+`java-suites` CI job (which compiles `consumers/` without TinkerPop) does not
+see TinkerPop. Two extra steps at the end of that job fetch the pinned jars and
+run the harness. It copies three `RowStore.openWithEdges` fixtures into TinkerGraph
+3.7.7, a test-oracle copy under rule E2. It then compares
+`Graph.from(seeds).hop(KNOWS)^k.count()` with
+`g.V(seeds).out("knows")^k.dedup().count()` for k = 1..3.
+
+- **Measured (JDK 28-ea+18, abi 0.12, avx512):** 25 checks green. All 9
+  counts are equal, the pinned fixture reproduces GraphHopTest's 19 and 29,
+  and the largest case is n=20000, k=3, with 873 distinct rows.
+- **Walks are not vertices:** the counts without `dedup()` are higher on every
+  fixture (e.g. 76 vs 55). Only the `dedup()` form may be lowered to a mask
+  chain, and the suite pins that it can tell the difference.
+- **Disable-verified:** with the chain one hop short, all 9 comparisons go red.
+- Dependencies: 19 jars pinned with sha256 in `deps.lock`. `fetch-deps.sh`
+  retries on HTTP 429 from Maven Central.
+- **Not done:** a provider strategy (A2). Nothing in `src/main` changed.
+
 ## 2026-09-22 — the Java gate is DISPATCHED, and one measurement stops being restated in eight places
 
 `ISS-LGJ-CONSUMERS-HAVE-NO-CI-LINE` is RESOLVED. `java-suites` in

.claude/board/PR_ARC_INVENTORY.md

@@ -8,6 +8,25 @@
 > anti-pattern the imported board rules name. Backfilled below in one
 > pass rather than left stale; PR #4 onward gets its entry at merge time.
 
+## PR (open) — A1 TinkerPop parity harness (branch `ccr-70e679fa-9qvl9y`)
+
+**Entry written at open.**
+
+- **Added:** `interop/tinkerpop/`, consisting of `TinkerParityTest`,
+  `deps.lock` (19 jars, sha256), `fetch-deps.sh`, `run.sh`, a README, and
+  gitignore lines for `lib/` and `out/`.
+- **Locked:** the lance-graph-java hop chain equals the Gremlin `dedup()`
+  count for k = 1..3 on three fixtures. The walk count without `dedup()`
+  differs, and the suite requires that difference.
+- **Deferred:** A2 (a `TraversalStrategy` that lowers compilable step runs
+  with a counted fallback).
+- **CI:** two steps at the end of `java-suites` fetch the sha256-pinned jars
+  and run the harness on the job's JDK 28 EA.
+- **Gates:** 25 checks green; disable run red-then-green. The core suite was
+  untouched (612 checks green before the change).
+- **Confidence:** high for parity on these fixtures. It says nothing yet about
+  the speed or allocation of any lowered path (A3).
+
 ## PR #81 — Panama × Valhalla as ONE production membrane (merged `07aa441`, branch `claude/great-pascal-k96kok`, head `1c66091`, 3 commits)
 
 **Entry written at merge**, not backfilled.

.github/workflows/lint.yml

@@ -343,3 +343,17 @@ jobs:
             exit 1
           fi
           echo "all four consumer suites passed"
+      # Quick win A1: TinkerGraph 3.7.7 vs mask hop chain parity. Kept out of
+      # `consumers/` because that tree is compiled above without TinkerPop on the
+      # classpath. `fetch-deps.sh` downloads 19 jars from Maven Central and
+      # refuses any whose sha256 differs from `deps.lock`; `run.sh` reads
+      # LGJ_JDK, not JAVA_HOME, so point it at the JDK 28 EA that setup-java
+      # installed in this job.
+      - name: Fetch TinkerPop jars (sha256-pinned)
+        working-directory: lance-graph-java/interop/tinkerpop
+        run: ./fetch-deps.sh
+      - name: Run the TinkerPop parity harness (A1)
+        working-directory: lance-graph-java/interop/tinkerpop
+        env:
+          LGJ_JDK: ${{ env.JAVA_HOME }}
+        run: ./run.sh

.gitignore

@@ -35,3 +35,7 @@ Cargo.lock.bak
 valhalla-lab/reproducers/*.so
 valhalla-lab/reproducers/r8_standalone
 valhalla-lab/reproducers/r8-ndarray/target/
+
+# interop/tinkerpop: fetched jars and compiled classes (fetch-deps.sh / run.sh)
+/interop/tinkerpop/lib/
+/interop/tinkerpop/out/

interop/tinkerpop/README.md

@@ -0,0 +1,33 @@
+# interop/tinkerpop: A1 parity harness
+
+One seeded graph, two engines, the same answers. The harness opens a
+`RowStore.openWithEdges` fixture and copies its edges into **TinkerGraph** 3.7.7,
+the TinkerPop line JanusGraph 1.x builds on. It then answers k-hop expansions
+from the same seeds twice:
+
+| Engine | Query |
+|---|---|
+| lance-graph-java | `Graph.from(seeds).hop(KNOWS)…count()`, a mask hop chain |
+| Gremlin | `g.V(seeds).out("knows")….dedup().count()` |
+
+```sh
+./fetch-deps.sh   # once: 19 jars from Maven Central, each sha256-checked against deps.lock
+./run.sh          # needs JDK 28 (LGJ_JDK, default /opt/jdks/jdk-28) and native/lgj-abi built --release
+```
+
+The harness lives outside `consumers/` on purpose. CI compiles every file under
+`consumers/` against `java/out` alone, and TinkerPop is not on that classpath.
+The `java-suites` job runs this harness in two separate steps at its end:
+`fetch-deps.sh`, then `run.sh` with `LGJ_JDK` set to the job's JDK 28 EA.
+
+## What it checks (25 checks, all green)
+
+- The copy has edges, so a match is not between two empty sets.
+- k = 1, 2, 3: the hop-chain count equals the Gremlin `dedup()` count on every fixture. The pinned fixture reproduces GraphHopTest's own 19 and 29.
+- Falsifier: hopping over a classid with no edges does not match Gremlin. A second falsifier, run by hand: with the chain one hop short, every comparison goes red.
+- Walks versus vertices: `out()…count()` without `dedup()` counts walks, which come out higher (for example 76 against 55). The suite requires at least one fixture where they differ, proving it can tell a bag from a set. Only the `dedup()` form may ever be lowered to a mask chain.
+
+## Rules it follows
+
+- The TinkerGraph copy reads facets one row at a time (`classidAt`, `payloadHi32At`, `payloadLow64At`). That is a scalar test oracle, which lance-graph-java licenses only in test code (rule E2). It is never an execution path.
+- Nothing here changes `src/main`. A provider strategy that rewrites Gremlin into mask chains is the next step (A2), and it must keep a counted, visible fallback for every traversal it does not lower.

interop/tinkerpop/deps.lock

@@ -0,0 +1,23 @@
+# TinkerGraph 3.7.7 runtime closure (the line JanusGraph 1.x builds on), resolved
+# once with `mvn dependency:copy-dependencies` and pinned here. fetch-deps.sh
+# downloads each jar from Maven Central and refuses any checksum mismatch.
+# Format: groupId:artifactId:version sha256
+com.carrotsearch:hppc:0.7.1 40d2a57f59e9eae7b018d3b4841954087ee40a5c1db6a54c3ea87742e3890391
+com.squareup:javapoet:1.13.0 4c7517e848a71b36d069d12bb3bf46a70fd4cda3105d822b0ed2e19c00b69291
+commons-beanutils:commons-beanutils:1.11.0 9e44ba68ec9a3f21286fa2a8bbb003b735c0f69101bb43144b79f4f8aaa74709
+commons-collections:commons-collections:3.2.2 eeeae917917144a68a741d4c0dff66aa5c5c5fd85593ff217bced3fc8ca783b8
+commons-io:commons-io:2.22.0 2b9a7b1f726fb86216dbd2c8321eabe0221dbd5b1be81c18e1cb53811b104758
+net.objecthunter:exp4j:0.4.8 271f7824ee8a3468257bc0613afdabb67597af8389317643fa806b983b7ecb27
+org.antlr:antlr4-runtime:4.9.1 a80502c3140ae7acfbb1f57847c8eb1c101461a969215ec38c3f1ebdff61ac29
+org.apache.commons:commons-configuration2:2.15.1 61ba5eaab8a17087042fe3d6811573bb85ccbdd6bd9d21ad464e6e20559dd1e8
+org.apache.commons:commons-lang3:3.20.0 69e5c9fa35da7a51a5fd2099dfe56a2d8d32cf233e2f6d770e796146440263f4
+org.apache.commons:commons-text:1.15.0 58d2da30f058512a1e7f914e39241deca4dff5c27a085b4ed2faa9e7208067f6
+org.apache.tinkerpop:gremlin-core:3.7.7 71cb32443cbdd9f925083be676a436c5ee8d14c097e71bd9fd73fd0a92d47bc8
+org.apache.tinkerpop:gremlin-language:3.7.7 44a1a80c046bc9a2b7832751039861ab5e69fe582d7564fde5c016ee6cb3724b
+org.apache.tinkerpop:gremlin-shaded:3.7.7 ed55f024b149cd34f8be54dd8b7d09df8ab56c3d1c1225d3ca9b2af569d392a1
+org.apache.tinkerpop:tinkergraph-gremlin:3.7.7 05648d37f7e1c11b16440033307b984aa1541fd75a26c15c95d8d7203c77ebcc
+org.javatuples:javatuples:1.2 2eda5b19d9820e1cc2f69fcd01639a715a673c11f8507e3d1ed593cf765d5e0a
+org.slf4j:jcl-over-slf4j:1.7.25 5e938457e79efcbfb3ab64bc29c43ec6c3b95fffcda3c155f4a86cc320c11e14
+org.slf4j:slf4j-api:1.7.36 d3ef575e3e4979678dc01bf1dcce51021493b4d11fb7f1be8ad982877c16a1c0
+org.slf4j:slf4j-nop:1.7.36 c214958b07816cb4412b30c7bdbd4308ffdc6ba2a83767b8f3a9229cbd9274d6
+org.yaml:snakeyaml:2.0 880c9d896e4b74a06c549c15ca496450165d6909fa15d7e662bee8f6a66d7afa

interop/tinkerpop/fetch-deps.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Download the pinned TinkerGraph closure into ./lib, verifying every checksum.
+set -euo pipefail
+cd "$(dirname "$0")"
+mkdir -p lib
+repo=https://repo1.maven.org/maven2
+grep -v '^#' deps.lock | while read -r coord sum; do
+  [ -z "$coord" ] && continue
+  IFS=: read -r g a v <<<"$coord"
+  jar="lib/$a-$v.jar"
+  if [ ! -f "$jar" ]; then
+    url="$repo/${g//./\/}/$a/$v/$a-$v.jar"
+    # Maven Central answers 429 under bursts; back off and retry a few times.
+    for delay in 0 2 4 8 16; do
+      sleep "$delay"
+      # Bound each attempt: a stalled transfer must time out so the loop advances.
+      if curl -fsS --connect-timeout 20 --max-time 120 -o "$jar.part" "$url"; then break; fi
+    done
+    mv "$jar.part" "$jar"
+  fi
+  echo "$sum  $jar" | sha256sum -c --quiet - || { echo "checksum mismatch: $jar" >&2; rm -f "$jar"; exit 1; }
+done
+echo "deps ok: $(grep -cv -e '^#' -e '^$' deps.lock) jars"

interop/tinkerpop/run.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Compile lance-graph-java, the graph consumer and this harness, then run it.
+# Needs: JDK 28 (LGJ_JDK, default /opt/jdks/jdk-28; JAVA_HOME is not used,
+# because it often points at an older JDK), the native library built in
+# native/lgj-abi (cargo build --release), and ./fetch-deps.sh run once.
+set -euo pipefail
+here="$(cd "$(dirname "$0")" && pwd)"
+root="$(cd "$here/../.." && pwd)"
+J="${LGJ_JDK:-/opt/jdks/jdk-28}/bin"
+SO="$root/native/lgj-abi/target/release/liblgj_abi.so"
+out="$here/out"
+rm -rf "$out" && mkdir -p "$out"
+# The classpath is exactly the jars pinned in deps.lock, never a lib/ wildcard:
+# a stale jar left in lib/ must not shadow a pinned one.
+cp=""
+while read -r coord _; do
+  [ -z "$coord" ] && continue
+  IFS=: read -r _ a v <<<"$coord"
+  jar="$here/lib/$a-$v.jar"
+  [ -f "$jar" ] || { echo "missing $jar; run ./fetch-deps.sh" >&2; exit 1; }
+  cp="$cp$jar:"
+done < <(grep -v '^#' "$here/deps.lock")
+if ! "$J/javac" --release 28 --enable-preview -d "$out" -cp "$cp" \
+  $(find "$root/java/src/main" "$root/java/src/test" "$root/consumers/graph/src/main" "$here/src" -name '*.java') \
+  >"$out/javac.log" 2>&1; then
+  cat "$out/javac.log" >&2
+  echo "compile failed" >&2
+  exit 1
+fi
+exec "$J/java" --enable-preview --enable-native-access=ALL-UNNAMED -Dlgj.library="$SO" \
+  -cp "$out:$cp" com.adaworldapi.interop.tinkerpop.TinkerParityTest

interop/tinkerpop/src/com/adaworldapi/interop/tinkerpop/TinkerParityTest.java

@@ -0,0 +1,181 @@
+package com.adaworldapi.interop.tinkerpop;
+
+import com.adaworldapi.graph.Edge;
+import com.adaworldapi.graph.Graph;
+import com.adaworldapi.lancegraph.Checks;
+import com.adaworldapi.lancegraph.FacetId;
+import com.adaworldapi.lancegraph.NativeRuntime;
+import com.adaworldapi.lancegraph.RowStore;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.structure.T;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+
+/**
+ * Quick win A1: one seeded graph, two engines, the same answers.
+ *
+ * <p>For each fixture, a {@link RowStore} is opened with {@link RowStore#openWithEdges}, and the
+ * same edges are copied into a TinkerGraph (TinkerPop's reference in-memory implementation). Then
+ * k-hop expansions from the same seed rows are answered twice:
+ *
+ * <ul>
+ *   <li>by lance-graph-java, as a mask hop chain ({@code Graph.from(seeds).hop(KNOWS)...count()});
+ *   <li>by Gremlin, {@code g.V(seeds).out("knows")...dedup().count()}.
+ * </ul>
+ *
+ * <p><strong>The copy into TinkerGraph is test code only.</strong> It reads every row's facets one
+ * at a time ({@code classidAt} / {@code payloadHi32At} / {@code payloadLow64At}), the same scalar
+ * decoding {@code GraphHopTest}'s reference oracles use. lance-graph-java licenses a per-row Java
+ * loop only as a test oracle (rule E2), never as an execution path.
+ *
+ * <p><strong>Walks are not vertices.</strong> {@code out().out().count()} counts walks, duplicates
+ * included; a mask hop chain counts distinct rows. Only the {@code dedup()} form is comparable. The
+ * suite records both, and requires at least one fixture where they differ, so the harness is shown
+ * able to tell them apart.
+ */
+public final class TinkerParityTest {
+
+    private TinkerParityTest() {}
+
+    private static final int FACETS_PER_ROW = 32;
+    private static final String KNOWS = "knows";
+    private static final int MAX_HOPS = 3;
+
+    private record Fixture(String name, long nRows, long seed, long gateMask, int radius,
+            long[] seeds) {}
+
+    private static long[] stride(int count, long step, long offset) {
+        long[] rows = new long[count];
+        for (int i = 0; i < count; i++) {
+            rows[i] = i * step + offset;
+        }
+        return rows;
+    }
+
+    private static List<Fixture> fixtures() {
+        return List.of(
+                // GraphHopTest's pinned fixture: 1 hop reaches 19 rows, 2 hops 29.
+                new Fixture("pinned n=2000", 2000L, 0xF00D_CAFEL, 0x0L, 25, stride(10, 37, 5)),
+                new Fixture("n=10000 r=60", 10_000L, 0x5EED_1234L, 0x0L, 60, stride(40, 241, 3)),
+                new Fixture("n=20000 r=120", 20_000L, 0xBADC_0FFEL, 0x0L, 120,
+                        stride(100, 197, 11)));
+    }
+
+    public static void main(String[] args) {
+        System.out.println("TinkerParityTest");
+        if (!NativeRuntime.isAvailable()) {
+            System.exit(Checks.reportUnavailable("TinkerParityTest"));
+        }
+        Checks c = new Checks("TinkerParityTest");
+        boolean[] walksDifferedSomewhere = {false};
+        for (Fixture f : fixtures()) {
+            runFixture(c, f, walksDifferedSomewhere);
+        }
+        c.section("walk count vs distinct count");
+        c.that("on at least one fixture, out()...count() (walks) differs from the dedup count,"
+                + " so the harness can tell bag from set", walksDifferedSomewhere[0]);
+        System.exit(c.report());
+    }
+
+    private static void runFixture(Checks c, Fixture f, boolean[] walksDiffered) {
+        c.section(f.name());
+        try (RowStore store = RowStore.openWithEdges(f.nRows(), f.seed(), Edge.KNOWS,
+                f.gateMask(), f.radius())) {
+            TinkerGraph tg = copyIntoTinkerGraph(store);
+            long edges = tg.traversal().E().count().next();
+            c.that("TinkerGraph copy has edges (" + edges + "), so the comparison is not vacuous",
+                    edges > 0);
+            GraphTraversalSource g = tg.traversal();
+            Object[] seedIds = boxed(f.seeds());
+
+            for (int k = 1; k <= MAX_HOPS; k++) {
+                long lance = lanceHopCount(store, f.seeds(), Edge.KNOWS, k);
+                long gremlinSet = outN(g.V(seedIds), k).dedup().count().next();
+                long gremlinWalks = outN(g.V(seedIds), k).count().next();
+                c.note(String.format("k=%d  lance-graph=%d  gremlin dedup=%d  gremlin walks=%d",
+                        k, lance, gremlinSet, gremlinWalks));
+                c.eq("k=" + k + ": lance-graph hop chain equals Gremlin dedup count",
+                        gremlinSet, lance);
+                c.that("k=" + k + ": walks >= distinct vertices", gremlinWalks >= gremlinSet);
+                if (gremlinWalks != gremlinSet) {
+                    walksDiffered[0] = true;
+                }
+                if (k == 1 && gremlinSet > 0) {
+                    // Falsifier: the comparison must be able to fail. Hopping a classid that
+                    // has no edges must not reproduce Gremlin's answer.
+                    long wrong = lanceHopCount(store, f.seeds(), Edge.KNOWS + 1, k);
+                    c.notEq("k=1: a hop over the wrong edge classid does NOT match Gremlin",
+                            gremlinSet, wrong);
+                }
+            }
+            tg.close();
+        } catch (Exception e) {
+            throw new IllegalStateException(e);
+        }
+    }
+
+    private static GraphTraversal<Vertex, Vertex> outN(GraphTraversal<Vertex, Vertex> t, int k) {
+        for (int i = 0; i < k; i++) {
+            t = t.out(KNOWS);
+        }
+        return t;
+    }
+
+    private static long lanceHopCount(RowStore store, long[] seeds, int edgeClassid, int hops) {
+        List<Graph> chain = new ArrayList<>();
+        try {
+            Graph g = Graph.open(store);
+            chain.add(g);
+            g = g.from(seeds);
+            chain.add(g);
+            for (int i = 0; i < hops; i++) {
+                g = g.hop(edgeClassid);
+                chain.add(g);
+            }
+            return g.count();
+        } finally {
+            for (Graph g : chain) {
+                g.close();
+            }
+        }
+    }
+
+    /** Test-oracle copy: every KNOWS facet becomes one Gremlin edge, duplicates kept. */
+    private static TinkerGraph copyIntoTinkerGraph(RowStore store) {
+        TinkerGraph tg = TinkerGraph.open();
+        long n = store.rowCount();
+        Vertex[] vs = new Vertex[(int) n];
+        for (long row = 0; row < n; row++) {
+            vs[(int) row] = tg.addVertex(T.id, row);
+        }
+        for (long row = 0; row < n; row++) {
+            for (int f = 0; f < FACETS_PER_ROW; f++) {
+                FacetId facet = FacetId.of(f);
+                if (store.classidAt(row, facet) != Edge.KNOWS) {
+                    continue;
+                }
+                if (store.payloadHi32At(row, facet) != 0) {
+                    continue;
+                }
+                long target = store.payloadLow64At(row, facet);
+                if (target >= 0 && target < n) {
+                    vs[(int) row].addEdge(KNOWS, vs[(int) target]);
+                }
+            }
+        }
+        return tg;
+    }
+
+    private static Object[] boxed(long[] rows) {
+        Object[] out = new Object[rows.length];
+        for (int i = 0; i < rows.length; i++) {
+            out[i] = rows[i];
+        }
+        return out;
+    }
+}