Skip to content

Commit 0136005

Browse files
authored
Merge pull request #89 from AdaWorldAPI/ccr-70e679fa-9qvl9y
Add interop/tinkerpop: Gremlin vs mask hop chain parity harness (A1)
2 parents 266fa70 + e87f406 commit 0136005

9 files changed

Lines changed: 349 additions & 0 deletions

File tree

‎.claude/board/LATEST_STATE.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,24 @@
1+
## 2026-10-09 — Gremlin and the mask hop chain agree (interop/tinkerpop, A1)
2+
3+
`interop/tinkerpop/` adds a parity harness, built outside `consumers/` so the
4+
`java-suites` CI job (which compiles `consumers/` without TinkerPop) does not
5+
see TinkerPop. Two extra steps at the end of that job fetch the pinned jars and
6+
run the harness. It copies three `RowStore.openWithEdges` fixtures into TinkerGraph
7+
3.7.7, a test-oracle copy under rule E2. It then compares
8+
`Graph.from(seeds).hop(KNOWS)^k.count()` with
9+
`g.V(seeds).out("knows")^k.dedup().count()` for k = 1..3.
10+
11+
- **Measured (JDK 28-ea+18, abi 0.12, avx512):** 25 checks green. All 9
12+
counts are equal, the pinned fixture reproduces GraphHopTest's 19 and 29,
13+
and the largest case is n=20000, k=3, with 873 distinct rows.
14+
- **Walks are not vertices:** the counts without `dedup()` are higher on every
15+
fixture (e.g. 76 vs 55). Only the `dedup()` form may be lowered to a mask
16+
chain, and the suite pins that it can tell the difference.
17+
- **Disable-verified:** with the chain one hop short, all 9 comparisons go red.
18+
- Dependencies: 19 jars pinned with sha256 in `deps.lock`. `fetch-deps.sh`
19+
retries on HTTP 429 from Maven Central.
20+
- **Not done:** a provider strategy (A2). Nothing in `src/main` changed.
21+
122
## 2026-09-22 — the Java gate is DISPATCHED, and one measurement stops being restated in eight places
223

324
`ISS-LGJ-CONSUMERS-HAVE-NO-CI-LINE` is RESOLVED. `java-suites` in

‎.claude/board/PR_ARC_INVENTORY.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,25 @@
88
> anti-pattern the imported board rules name. Backfilled below in one
99
> pass rather than left stale; PR #4 onward gets its entry at merge time.
1010
11+
## PR (open) — A1 TinkerPop parity harness (branch `ccr-70e679fa-9qvl9y`)
12+
13+
**Entry written at open.**
14+
15+
- **Added:** `interop/tinkerpop/`, consisting of `TinkerParityTest`,
16+
`deps.lock` (19 jars, sha256), `fetch-deps.sh`, `run.sh`, a README, and
17+
gitignore lines for `lib/` and `out/`.
18+
- **Locked:** the lance-graph-java hop chain equals the Gremlin `dedup()`
19+
count for k = 1..3 on three fixtures. The walk count without `dedup()`
20+
differs, and the suite requires that difference.
21+
- **Deferred:** A2 (a `TraversalStrategy` that lowers compilable step runs
22+
with a counted fallback).
23+
- **CI:** two steps at the end of `java-suites` fetch the sha256-pinned jars
24+
and run the harness on the job's JDK 28 EA.
25+
- **Gates:** 25 checks green; disable run red-then-green. The core suite was
26+
untouched (612 checks green before the change).
27+
- **Confidence:** high for parity on these fixtures. It says nothing yet about
28+
the speed or allocation of any lowered path (A3).
29+
1130
## PR #81 — Panama × Valhalla as ONE production membrane (merged `07aa441`, branch `claude/great-pascal-k96kok`, head `1c66091`, 3 commits)
1231

1332
**Entry written at merge**, not backfilled.

‎.github/workflows/lint.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -343,3 +343,17 @@ jobs:
343343
exit 1
344344
fi
345345
echo "all four consumer suites passed"
346+
# Quick win A1: TinkerGraph 3.7.7 vs mask hop chain parity. Kept out of
347+
# `consumers/` because that tree is compiled above without TinkerPop on the
348+
# classpath. `fetch-deps.sh` downloads 19 jars from Maven Central and
349+
# refuses any whose sha256 differs from `deps.lock`; `run.sh` reads
350+
# LGJ_JDK, not JAVA_HOME, so point it at the JDK 28 EA that setup-java
351+
# installed in this job.
352+
- name: Fetch TinkerPop jars (sha256-pinned)
353+
working-directory: lance-graph-java/interop/tinkerpop
354+
run: ./fetch-deps.sh
355+
- name: Run the TinkerPop parity harness (A1)
356+
working-directory: lance-graph-java/interop/tinkerpop
357+
env:
358+
LGJ_JDK: ${{ env.JAVA_HOME }}
359+
run: ./run.sh

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,7 @@ Cargo.lock.bak
3535
valhalla-lab/reproducers/*.so
3636
valhalla-lab/reproducers/r8_standalone
3737
valhalla-lab/reproducers/r8-ndarray/target/
38+
39+
# interop/tinkerpop: fetched jars and compiled classes (fetch-deps.sh / run.sh)
40+
/interop/tinkerpop/lib/
41+
/interop/tinkerpop/out/

‎interop/tinkerpop/README.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# interop/tinkerpop: A1 parity harness
2+
3+
One seeded graph, two engines, the same answers. The harness opens a
4+
`RowStore.openWithEdges` fixture and copies its edges into **TinkerGraph** 3.7.7,
5+
the TinkerPop line JanusGraph 1.x builds on. It then answers k-hop expansions
6+
from the same seeds twice:
7+
8+
| Engine | Query |
9+
|---|---|
10+
| lance-graph-java | `Graph.from(seeds).hop(KNOWS)…count()`, a mask hop chain |
11+
| Gremlin | `g.V(seeds).out("knows")….dedup().count()` |
12+
13+
```sh
14+
./fetch-deps.sh # once: 19 jars from Maven Central, each sha256-checked against deps.lock
15+
./run.sh # needs JDK 28 (LGJ_JDK, default /opt/jdks/jdk-28) and native/lgj-abi built --release
16+
```
17+
18+
The harness lives outside `consumers/` on purpose. CI compiles every file under
19+
`consumers/` against `java/out` alone, and TinkerPop is not on that classpath.
20+
The `java-suites` job runs this harness in two separate steps at its end:
21+
`fetch-deps.sh`, then `run.sh` with `LGJ_JDK` set to the job's JDK 28 EA.
22+
23+
## What it checks (25 checks, all green)
24+
25+
- The copy has edges, so a match is not between two empty sets.
26+
- k = 1, 2, 3: the hop-chain count equals the Gremlin `dedup()` count on every fixture. The pinned fixture reproduces GraphHopTest's own 19 and 29.
27+
- Falsifier: hopping over a classid with no edges does not match Gremlin. A second falsifier, run by hand: with the chain one hop short, every comparison goes red.
28+
- Walks versus vertices: `out()…count()` without `dedup()` counts walks, which come out higher (for example 76 against 55). The suite requires at least one fixture where they differ, proving it can tell a bag from a set. Only the `dedup()` form may ever be lowered to a mask chain.
29+
30+
## Rules it follows
31+
32+
- The TinkerGraph copy reads facets one row at a time (`classidAt`, `payloadHi32At`, `payloadLow64At`). That is a scalar test oracle, which lance-graph-java licenses only in test code (rule E2). It is never an execution path.
33+
- Nothing here changes `src/main`. A provider strategy that rewrites Gremlin into mask chains is the next step (A2), and it must keep a counted, visible fallback for every traversal it does not lower.

‎interop/tinkerpop/deps.lock‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# TinkerGraph 3.7.7 runtime closure (the line JanusGraph 1.x builds on), resolved
2+
# once with `mvn dependency:copy-dependencies` and pinned here. fetch-deps.sh
3+
# downloads each jar from Maven Central and refuses any checksum mismatch.
4+
# Format: groupId:artifactId:version sha256
5+
com.carrotsearch:hppc:0.7.1 40d2a57f59e9eae7b018d3b4841954087ee40a5c1db6a54c3ea87742e3890391
6+
com.squareup:javapoet:1.13.0 4c7517e848a71b36d069d12bb3bf46a70fd4cda3105d822b0ed2e19c00b69291
7+
commons-beanutils:commons-beanutils:1.11.0 9e44ba68ec9a3f21286fa2a8bbb003b735c0f69101bb43144b79f4f8aaa74709
8+
commons-collections:commons-collections:3.2.2 eeeae917917144a68a741d4c0dff66aa5c5c5fd85593ff217bced3fc8ca783b8
9+
commons-io:commons-io:2.22.0 2b9a7b1f726fb86216dbd2c8321eabe0221dbd5b1be81c18e1cb53811b104758
10+
net.objecthunter:exp4j:0.4.8 271f7824ee8a3468257bc0613afdabb67597af8389317643fa806b983b7ecb27
11+
org.antlr:antlr4-runtime:4.9.1 a80502c3140ae7acfbb1f57847c8eb1c101461a969215ec38c3f1ebdff61ac29
12+
org.apache.commons:commons-configuration2:2.15.1 61ba5eaab8a17087042fe3d6811573bb85ccbdd6bd9d21ad464e6e20559dd1e8
13+
org.apache.commons:commons-lang3:3.20.0 69e5c9fa35da7a51a5fd2099dfe56a2d8d32cf233e2f6d770e796146440263f4
14+
org.apache.commons:commons-text:1.15.0 58d2da30f058512a1e7f914e39241deca4dff5c27a085b4ed2faa9e7208067f6
15+
org.apache.tinkerpop:gremlin-core:3.7.7 71cb32443cbdd9f925083be676a436c5ee8d14c097e71bd9fd73fd0a92d47bc8
16+
org.apache.tinkerpop:gremlin-language:3.7.7 44a1a80c046bc9a2b7832751039861ab5e69fe582d7564fde5c016ee6cb3724b
17+
org.apache.tinkerpop:gremlin-shaded:3.7.7 ed55f024b149cd34f8be54dd8b7d09df8ab56c3d1c1225d3ca9b2af569d392a1
18+
org.apache.tinkerpop:tinkergraph-gremlin:3.7.7 05648d37f7e1c11b16440033307b984aa1541fd75a26c15c95d8d7203c77ebcc
19+
org.javatuples:javatuples:1.2 2eda5b19d9820e1cc2f69fcd01639a715a673c11f8507e3d1ed593cf765d5e0a
20+
org.slf4j:jcl-over-slf4j:1.7.25 5e938457e79efcbfb3ab64bc29c43ec6c3b95fffcda3c155f4a86cc320c11e14
21+
org.slf4j:slf4j-api:1.7.36 d3ef575e3e4979678dc01bf1dcce51021493b4d11fb7f1be8ad982877c16a1c0
22+
org.slf4j:slf4j-nop:1.7.36 c214958b07816cb4412b30c7bdbd4308ffdc6ba2a83767b8f3a9229cbd9274d6
23+
org.yaml:snakeyaml:2.0 880c9d896e4b74a06c549c15ca496450165d6909fa15d7e662bee8f6a66d7afa

‎interop/tinkerpop/fetch-deps.sh‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/usr/bin/env bash
2+
# Download the pinned TinkerGraph closure into ./lib, verifying every checksum.
3+
set -euo pipefail
4+
cd "$(dirname "$0")"
5+
mkdir -p lib
6+
repo=https://repo1.maven.org/maven2
7+
grep -v '^#' deps.lock | while read -r coord sum; do
8+
[ -z "$coord" ] && continue
9+
IFS=: read -r g a v <<<"$coord"
10+
jar="lib/$a-$v.jar"
11+
if [ ! -f "$jar" ]; then
12+
url="$repo/${g//./\/}/$a/$v/$a-$v.jar"
13+
# Maven Central answers 429 under bursts; back off and retry a few times.
14+
for delay in 0 2 4 8 16; do
15+
sleep "$delay"
16+
# Bound each attempt: a stalled transfer must time out so the loop advances.
17+
if curl -fsS --connect-timeout 20 --max-time 120 -o "$jar.part" "$url"; then break; fi
18+
done
19+
mv "$jar.part" "$jar"
20+
fi
21+
echo "$sum $jar" | sha256sum -c --quiet - || { echo "checksum mismatch: $jar" >&2; rm -f "$jar"; exit 1; }
22+
done
23+
echo "deps ok: $(grep -cv -e '^#' -e '^$' deps.lock) jars"

‎interop/tinkerpop/run.sh‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/usr/bin/env bash
2+
# Compile lance-graph-java, the graph consumer and this harness, then run it.
3+
# Needs: JDK 28 (LGJ_JDK, default /opt/jdks/jdk-28; JAVA_HOME is not used,
4+
# because it often points at an older JDK), the native library built in
5+
# native/lgj-abi (cargo build --release), and ./fetch-deps.sh run once.
6+
set -euo pipefail
7+
here="$(cd "$(dirname "$0")" && pwd)"
8+
root="$(cd "$here/../.." && pwd)"
9+
J="${LGJ_JDK:-/opt/jdks/jdk-28}/bin"
10+
SO="$root/native/lgj-abi/target/release/liblgj_abi.so"
11+
out="$here/out"
12+
rm -rf "$out" && mkdir -p "$out"
13+
# The classpath is exactly the jars pinned in deps.lock, never a lib/ wildcard:
14+
# a stale jar left in lib/ must not shadow a pinned one.
15+
cp=""
16+
while read -r coord _; do
17+
[ -z "$coord" ] && continue
18+
IFS=: read -r _ a v <<<"$coord"
19+
jar="$here/lib/$a-$v.jar"
20+
[ -f "$jar" ] || { echo "missing $jar; run ./fetch-deps.sh" >&2; exit 1; }
21+
cp="$cp$jar:"
22+
done < <(grep -v '^#' "$here/deps.lock")
23+
if ! "$J/javac" --release 28 --enable-preview -d "$out" -cp "$cp" \
24+
$(find "$root/java/src/main" "$root/java/src/test" "$root/consumers/graph/src/main" "$here/src" -name '*.java') \
25+
>"$out/javac.log" 2>&1; then
26+
cat "$out/javac.log" >&2
27+
echo "compile failed" >&2
28+
exit 1
29+
fi
30+
exec "$J/java" --enable-preview --enable-native-access=ALL-UNNAMED -Dlgj.library="$SO" \
31+
-cp "$out:$cp" com.adaworldapi.interop.tinkerpop.TinkerParityTest
Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
package com.adaworldapi.interop.tinkerpop;
2+
3+
import com.adaworldapi.graph.Edge;
4+
import com.adaworldapi.graph.Graph;
5+
import com.adaworldapi.lancegraph.Checks;
6+
import com.adaworldapi.lancegraph.FacetId;
7+
import com.adaworldapi.lancegraph.NativeRuntime;
8+
import com.adaworldapi.lancegraph.RowStore;
9+
10+
import java.util.ArrayList;
11+
import java.util.List;
12+
13+
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
14+
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
15+
import org.apache.tinkerpop.gremlin.structure.T;
16+
import org.apache.tinkerpop.gremlin.structure.Vertex;
17+
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
18+
19+
/**
20+
* Quick win A1: one seeded graph, two engines, the same answers.
21+
*
22+
* <p>For each fixture, a {@link RowStore} is opened with {@link RowStore#openWithEdges}, and the
23+
* same edges are copied into a TinkerGraph (TinkerPop's reference in-memory implementation). Then
24+
* k-hop expansions from the same seed rows are answered twice:
25+
*
26+
* <ul>
27+
* <li>by lance-graph-java, as a mask hop chain ({@code Graph.from(seeds).hop(KNOWS)...count()});
28+
* <li>by Gremlin, {@code g.V(seeds).out("knows")...dedup().count()}.
29+
* </ul>
30+
*
31+
* <p><strong>The copy into TinkerGraph is test code only.</strong> It reads every row's facets one
32+
* at a time ({@code classidAt} / {@code payloadHi32At} / {@code payloadLow64At}), the same scalar
33+
* decoding {@code GraphHopTest}'s reference oracles use. lance-graph-java licenses a per-row Java
34+
* loop only as a test oracle (rule E2), never as an execution path.
35+
*
36+
* <p><strong>Walks are not vertices.</strong> {@code out().out().count()} counts walks, duplicates
37+
* included; a mask hop chain counts distinct rows. Only the {@code dedup()} form is comparable. The
38+
* suite records both, and requires at least one fixture where they differ, so the harness is shown
39+
* able to tell them apart.
40+
*/
41+
public final class TinkerParityTest {
42+
43+
private TinkerParityTest() {}
44+
45+
private static final int FACETS_PER_ROW = 32;
46+
private static final String KNOWS = "knows";
47+
private static final int MAX_HOPS = 3;
48+
49+
private record Fixture(String name, long nRows, long seed, long gateMask, int radius,
50+
long[] seeds) {}
51+
52+
private static long[] stride(int count, long step, long offset) {
53+
long[] rows = new long[count];
54+
for (int i = 0; i < count; i++) {
55+
rows[i] = i * step + offset;
56+
}
57+
return rows;
58+
}
59+
60+
private static List<Fixture> fixtures() {
61+
return List.of(
62+
// GraphHopTest's pinned fixture: 1 hop reaches 19 rows, 2 hops 29.
63+
new Fixture("pinned n=2000", 2000L, 0xF00D_CAFEL, 0x0L, 25, stride(10, 37, 5)),
64+
new Fixture("n=10000 r=60", 10_000L, 0x5EED_1234L, 0x0L, 60, stride(40, 241, 3)),
65+
new Fixture("n=20000 r=120", 20_000L, 0xBADC_0FFEL, 0x0L, 120,
66+
stride(100, 197, 11)));
67+
}
68+
69+
public static void main(String[] args) {
70+
System.out.println("TinkerParityTest");
71+
if (!NativeRuntime.isAvailable()) {
72+
System.exit(Checks.reportUnavailable("TinkerParityTest"));
73+
}
74+
Checks c = new Checks("TinkerParityTest");
75+
boolean[] walksDifferedSomewhere = {false};
76+
for (Fixture f : fixtures()) {
77+
runFixture(c, f, walksDifferedSomewhere);
78+
}
79+
c.section("walk count vs distinct count");
80+
c.that("on at least one fixture, out()...count() (walks) differs from the dedup count,"
81+
+ " so the harness can tell bag from set", walksDifferedSomewhere[0]);
82+
System.exit(c.report());
83+
}
84+
85+
private static void runFixture(Checks c, Fixture f, boolean[] walksDiffered) {
86+
c.section(f.name());
87+
try (RowStore store = RowStore.openWithEdges(f.nRows(), f.seed(), Edge.KNOWS,
88+
f.gateMask(), f.radius())) {
89+
TinkerGraph tg = copyIntoTinkerGraph(store);
90+
long edges = tg.traversal().E().count().next();
91+
c.that("TinkerGraph copy has edges (" + edges + "), so the comparison is not vacuous",
92+
edges > 0);
93+
GraphTraversalSource g = tg.traversal();
94+
Object[] seedIds = boxed(f.seeds());
95+
96+
for (int k = 1; k <= MAX_HOPS; k++) {
97+
long lance = lanceHopCount(store, f.seeds(), Edge.KNOWS, k);
98+
long gremlinSet = outN(g.V(seedIds), k).dedup().count().next();
99+
long gremlinWalks = outN(g.V(seedIds), k).count().next();
100+
c.note(String.format("k=%d lance-graph=%d gremlin dedup=%d gremlin walks=%d",
101+
k, lance, gremlinSet, gremlinWalks));
102+
c.eq("k=" + k + ": lance-graph hop chain equals Gremlin dedup count",
103+
gremlinSet, lance);
104+
c.that("k=" + k + ": walks >= distinct vertices", gremlinWalks >= gremlinSet);
105+
if (gremlinWalks != gremlinSet) {
106+
walksDiffered[0] = true;
107+
}
108+
if (k == 1 && gremlinSet > 0) {
109+
// Falsifier: the comparison must be able to fail. Hopping a classid that
110+
// has no edges must not reproduce Gremlin's answer.
111+
long wrong = lanceHopCount(store, f.seeds(), Edge.KNOWS + 1, k);
112+
c.notEq("k=1: a hop over the wrong edge classid does NOT match Gremlin",
113+
gremlinSet, wrong);
114+
}
115+
}
116+
tg.close();
117+
} catch (Exception e) {
118+
throw new IllegalStateException(e);
119+
}
120+
}
121+
122+
private static GraphTraversal<Vertex, Vertex> outN(GraphTraversal<Vertex, Vertex> t, int k) {
123+
for (int i = 0; i < k; i++) {
124+
t = t.out(KNOWS);
125+
}
126+
return t;
127+
}
128+
129+
private static long lanceHopCount(RowStore store, long[] seeds, int edgeClassid, int hops) {
130+
List<Graph> chain = new ArrayList<>();
131+
try {
132+
Graph g = Graph.open(store);
133+
chain.add(g);
134+
g = g.from(seeds);
135+
chain.add(g);
136+
for (int i = 0; i < hops; i++) {
137+
g = g.hop(edgeClassid);
138+
chain.add(g);
139+
}
140+
return g.count();
141+
} finally {
142+
for (Graph g : chain) {
143+
g.close();
144+
}
145+
}
146+
}
147+
148+
/** Test-oracle copy: every KNOWS facet becomes one Gremlin edge, duplicates kept. */
149+
private static TinkerGraph copyIntoTinkerGraph(RowStore store) {
150+
TinkerGraph tg = TinkerGraph.open();
151+
long n = store.rowCount();
152+
Vertex[] vs = new Vertex[(int) n];
153+
for (long row = 0; row < n; row++) {
154+
vs[(int) row] = tg.addVertex(T.id, row);
155+
}
156+
for (long row = 0; row < n; row++) {
157+
for (int f = 0; f < FACETS_PER_ROW; f++) {
158+
FacetId facet = FacetId.of(f);
159+
if (store.classidAt(row, facet) != Edge.KNOWS) {
160+
continue;
161+
}
162+
if (store.payloadHi32At(row, facet) != 0) {
163+
continue;
164+
}
165+
long target = store.payloadLow64At(row, facet);
166+
if (target >= 0 && target < n) {
167+
vs[(int) row].addEdge(KNOWS, vs[(int) target]);
168+
}
169+
}
170+
}
171+
return tg;
172+
}
173+
174+
private static Object[] boxed(long[] rows) {
175+
Object[] out = new Object[rows.length];
176+
for (int i = 0; i < rows.length; i++) {
177+
out[i] = rows[i];
178+
}
179+
return out;
180+
}
181+
}

0 commit comments

Comments
 (0)