interop/tinkerpop: A2 lowering strategy (Gremlin → mask hop chain)

.claude/board/LATEST_STATE.md

@@ -1,3 +1,18 @@
+## 2026-10-09 — Gremlin traversals lowered onto the mask hop chain (interop/tinkerpop, A2)
+
+`LanceHopStrategy` is a TinkerPop provider strategy. It rewrites
+`V(ids).out(label)^k.dedup().count()` into one `LanceHopCountStep` answered by the
+lance-graph-java mask hop chain, and leaves everything else to Gremlin with a
+counted fallback and a reason.
+
+- **Measured:** 55 checks green. Lowered results equal plain Gremlin for k = 1..3,
+  through both `out()^k` and `repeat().times(k)`. Against an empty TinkerGraph the
+  lowered traversal still returns the row store's count, which proves where the
+  answer comes from.
+- **Found:** Gremlin turns `out().count()` into `outE().count()` before provider
+  strategies run, so a walk count needs `barrier()` to reach the `dedup()` guard.
+  That case is now a test, and removing the guard turns it red.
+
 ## 2026-10-09 — Gremlin and the mask hop chain agree (interop/tinkerpop, A1)
 
 `interop/tinkerpop/` adds a parity harness, built outside `consumers/` so the

.claude/board/PR_ARC_INVENTORY.md

@@ -8,10 +8,29 @@
 > anti-pattern the imported board rules name. Backfilled below in one
 > pass rather than left stale; PR #4 onward gets its entry at merge time.
 
-## PR (open) — A1 TinkerPop parity harness (branch `ccr-70e679fa-9qvl9y`)
+## PR (open) — A2 TinkerPop lowering strategy (branch `ccr-70e679fa-9qvl9y`)
 
 **Entry written at open.**
 
+- **Added:** `interop/tinkerpop/LanceHopStrategy` (a TinkerPop
+  `ProviderOptimizationStrategy`), `LanceHopCountStep`, `LoweringStrategyTest`;
+  `run.sh` now runs both mains and fails if either fails.
+- **Locked:** only `V(ids).out(label)^k.dedup().count()` is lowered (k >= 1, one
+  mapped label per hop, numeric row ids within the store, no `has()` or `as()`).
+  Every other root traversal is counted as a fallback with a reason and answers
+  exactly as Gremlin.
+- **Gates:** 55 checks green (plus A1's 25). Disable runs: the `dedup()` guard
+  removed turns 6 checks red (a walk count through `barrier()` gets lowered to a
+  distinct count); a lowered step answering 0 turns 7 red.
+- **Deferred:** A3 (measure lowered vs plain Gremlin), A5 (Gremlin Server
+  endpoint).
+- **Confidence:** high for correctness of the lowered shape on this fixture.
+  Nothing yet about speed.
+
+## PR #89 — A1 TinkerPop parity harness (merged `0136005`, branch `ccr-70e679fa-9qvl9y`)
+
+**Entry written at open; merge recorded 2026-10-09.**
+
 - **Added:** `interop/tinkerpop/`, consisting of `TinkerParityTest`,
   `deps.lock` (19 jars, sha256), `fetch-deps.sh`, `run.sh`, a README, and
   gitignore lines for `lib/` and `out/`.

interop/tinkerpop/README.md

@@ -1,4 +1,4 @@
-# interop/tinkerpop: A1 parity harness
+# interop/tinkerpop: A1 parity harness and A2 lowering strategy
 
 One seeded graph, two engines, the same answers. The harness opens a
 `RowStore.openWithEdges` fixture and copies its edges into **TinkerGraph** 3.7.7,
@@ -30,4 +30,46 @@ The `java-suites` job runs this harness in two separate steps at its end:
 ## Rules it follows
 
 - The TinkerGraph copy reads facets one row at a time (`classidAt`, `payloadHi32At`, `payloadLow64At`). That is a scalar test oracle, which lance-graph-java licenses only in test code (rule E2). It is never an execution path.
-- Nothing here changes `src/main`. A provider strategy that rewrites Gremlin into mask chains is the next step (A2), and it must keep a counted, visible fallback for every traversal it does not lower.
+- Nothing here changes `src/main`.
+
+## A2: `LanceHopStrategy` (55 checks, all green)
+
+A TinkerPop provider strategy. Register it on a traversal source:
+
+```java
+GraphTraversalSource g = graph.traversal()
+        .withStrategies(new LanceHopStrategy(store, Map.of("knows", Edge.KNOWS)));
+long n = g.V(seeds).out("knows").out("knows").dedup().count().next();
+```
+
+It lowers exactly one shape, `V(ids).out(label)^k.dedup().count()` with k >= 1, onto
+the mask hop chain. The whole traversal becomes one `LanceHopCountStep`, which
+emits the count when iterated. `repeat(out(label)).times(k)` reaches the same
+shape because TinkerPop unrolls it first. Vertex ids are row numbers, and each
+edge label maps to an edge classid.
+
+Everything else stays with Gremlin, and each such root traversal is counted in
+`fellBack()`, with the reason in `lastFallbackReason()`:
+
+- a walk count (no `dedup()`), which is a bag, not a set;
+- `has()`, `in()`, `both()`, an unmapped label, more than one label per hop;
+- `V()` without ids, a non-numeric id, or an id outside the row store;
+- any `as()` label.
+
+`LoweringStrategyTest` checks four things:
+
+- **Parity:** lowered equals plain Gremlin for k = 1..3, through both `out()^k` and `repeat()`, and the traversal really ran as one `LanceHopCountStep`.
+- **Provenance:** against an *empty* TinkerGraph the lowered traversal still returns the row store's count, while plain Gremlin returns 0. The answer comes from the mask chain, not from Gremlin.
+- **Fallbacks:** each excluded shape is counted once and answers exactly as Gremlin.
+- **Counters:** the lowered and fallback counters move by one per root traversal.
+
+Disable runs, each red then green:
+- remove the `dedup()` guard: the walk count through `barrier()` gets lowered and gives the distinct count instead (6 red);
+- make the lowered step return 0: every parity check and the provenance check go red (7 red).
+
+Gremlin rewrites `out().count()` into `outE().count()` before provider strategies
+run, so a plain walk count never reaches the `dedup()` guard. The `barrier()`
+case exists to exercise that guard.
+
+The strategy reads the row store it was given, not the graph the traversal
+source wraps. Keeping both consistent is the caller's job.

interop/tinkerpop/run.sh

@@ -27,5 +27,13 @@ if ! "$J/javac" --release 28 --enable-preview -d "$out" -cp "$cp" \
   echo "compile failed" >&2
   exit 1
 fi
-exec "$J/java" --enable-preview --enable-native-access=ALL-UNNAMED -Dlgj.library="$SO" \
-  -cp "$out:$cp" com.adaworldapi.interop.tinkerpop.TinkerParityTest
+# Run every main, then fail if any failed: one red suite must not hide the next.
+failed=""
+for main in TinkerParityTest LoweringStrategyTest; do
+  "$J/java" --enable-preview --enable-native-access=ALL-UNNAMED -Dlgj.library="$SO" \
+    -cp "$out:$cp" "com.adaworldapi.interop.tinkerpop.$main" || failed="$failed $main"
+done
+if [ -n "$failed" ]; then
+  echo "FAILED:$failed" >&2
+  exit 1
+fi

interop/tinkerpop/src/com/adaworldapi/interop/tinkerpop/LanceHopCountStep.java

@@ -0,0 +1,79 @@
+package com.adaworldapi.interop.tinkerpop;
+
+import com.adaworldapi.graph.Graph;
+import com.adaworldapi.lancegraph.RowStore;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
+import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
+import org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep;
+import org.apache.tinkerpop.gremlin.process.traversal.util.FastNoSuchElementException;
+
+/**
+ * The step {@link LanceHopStrategy} puts in place of {@code V(ids).out(l)^k.dedup().count()}.
+ * It emits one value: the number of distinct rows the mask hop chain reaches from the seeds.
+ *
+ * <p>The answer is computed when the traversal is first iterated, not when the strategy runs,
+ * so building a traversal costs nothing until it is used.
+ */
+final class LanceHopCountStep<S> extends AbstractStep<S, Long> {
+
+    private final RowStore store;
+    private final long[] seeds;
+    private final int[] hops;
+    private boolean done;
+
+    LanceHopCountStep(Traversal.Admin<?, ?> traversal, RowStore store, long[] seeds, int[] hops) {
+        super(traversal);
+        this.store = store;
+        this.seeds = seeds.clone();
+        this.hops = hops.clone();
+    }
+
+    @Override
+    protected Traverser.Admin<Long> processNextStart() {
+        if (done) {
+            throw FastNoSuchElementException.instance();
+        }
+        done = true;
+        // The generator is typed by the step the traverser starts at; this step starts the
+        // lowered traversal and produces the Long it carries.
+        @SuppressWarnings({"rawtypes", "unchecked"})
+        Traverser.Admin<Long> t = getTraversal().getTraverserGenerator()
+                .generate(count(), (org.apache.tinkerpop.gremlin.process.traversal.Step) this, 1L);
+        return t;
+    }
+
+    private long count() {
+        List<Graph> chain = new ArrayList<>();
+        try {
+            Graph g = Graph.open(store);
+            chain.add(g);
+            g = g.from(seeds);
+            chain.add(g);
+            for (int classid : hops) {
+                g = g.hop(classid);
+                chain.add(g);
+            }
+            return g.count();
+        } finally {
+            for (Graph g : chain) {
+                g.close();
+            }
+        }
+    }
+
+    @Override
+    public void reset() {
+        super.reset();
+        done = false;
+    }
+
+    @Override
+    public String toString() {
+        return "LanceHopCountStep(seeds=" + seeds.length + ", hops=" + Arrays.toString(hops) + ")";
+    }
+}

interop/tinkerpop/src/com/adaworldapi/interop/tinkerpop/LanceHopStrategy.java

@@ -0,0 +1,189 @@
+package com.adaworldapi.interop.tinkerpop;
+
+import com.adaworldapi.lancegraph.RowStore;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.tinkerpop.gremlin.process.traversal.Step;
+import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
+import org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy.ProviderOptimizationStrategy;
+import org.apache.tinkerpop.gremlin.process.traversal.step.HasContainerHolder;
+import org.apache.tinkerpop.gremlin.process.traversal.step.filter.DedupGlobalStep;
+import org.apache.tinkerpop.gremlin.process.traversal.step.map.CountGlobalStep;
+import org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep;
+import org.apache.tinkerpop.gremlin.process.traversal.step.map.VertexStep;
+import org.apache.tinkerpop.gremlin.process.traversal.step.util.EmptyStep;
+import org.apache.tinkerpop.gremlin.process.traversal.step.map.NoOpBarrierStep;
+import org.apache.tinkerpop.gremlin.process.traversal.strategy.AbstractTraversalStrategy;
+import org.apache.tinkerpop.gremlin.process.traversal.util.TraversalHelper;
+import org.apache.tinkerpop.gremlin.structure.Direction;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
+
+/**
+ * Quick win A2: a TinkerPop provider strategy that lowers one traversal shape onto the
+ * lance-graph-java mask hop chain, and leaves every other traversal to Gremlin.
+ *
+ * <p>The lowered shape is exactly
+ *
+ * <pre>{@code g.V(ids).out(label)^k.dedup().count()}</pre>
+ *
+ * with k &gt;= 1, one mapped edge label per hop, and no {@code has()} folded into the start
+ * step. {@code repeat(out(label)).times(k)} reaches the same shape, because TinkerPop's
+ * {@code RepeatUnrollStrategy} unrolls it before provider strategies run. A1 proved the
+ * {@code dedup()} form equal to the mask chain; the form without {@code dedup()} counts walks,
+ * not vertices, and is never lowered.
+ *
+ * <p><strong>Every decision is counted.</strong> {@link #lowered()} counts traversals replaced
+ * by a {@link LanceHopCountStep}; {@link #fellBack()} counts root traversals left to Gremlin,
+ * with the first reason kept in {@link #lastFallbackReason()}. A fallback is never an error:
+ * Gremlin still answers, so a caller only loses speed, never correctness.
+ *
+ * <p>The strategy reads the row store it was given; the graph the traversal source wraps is not
+ * consulted for a lowered traversal. Keeping the two consistent is the caller's job (in the
+ * harness both are built from the same fixture). Vertex ids are row numbers.
+ */
+public final class LanceHopStrategy
+        extends AbstractTraversalStrategy<ProviderOptimizationStrategy>
+        implements ProviderOptimizationStrategy {
+
+    private final RowStore store;
+    private final Map<String, Integer> edgeClassids;
+    private final AtomicLong lowered = new AtomicLong();
+    private final AtomicLong fellBack = new AtomicLong();
+    private volatile String lastFallbackReason = "";
+
+    /**
+     * @param store the row store lowered traversals are answered from
+     * @param edgeClassids Gremlin edge label to lance-graph edge classid; a label not in the map
+     *     is not lowered
+     */
+    public LanceHopStrategy(RowStore store, Map<String, Integer> edgeClassids) {
+        this.store = store;
+        this.edgeClassids = Map.copyOf(edgeClassids);
+    }
+
+    /** Number of traversals answered by the mask hop chain. */
+    public long lowered() {
+        return lowered.get();
+    }
+
+    /** Number of root traversals left to Gremlin. */
+    public long fellBack() {
+        return fellBack.get();
+    }
+
+    /** Why the most recent fallback happened; empty if none has. */
+    public String lastFallbackReason() {
+        return lastFallbackReason;
+    }
+
+    @Override
+    public void apply(Traversal.Admin<?, ?> traversal) {
+        // Child traversals (inside by(), repeat() bodies, ...) are not candidates; only the
+        // root is, and only the root is counted, so one query counts once.
+        if (!(traversal.getParent() instanceof EmptyStep)) {
+            return;
+        }
+        String reason = plan(traversal);
+        if (reason != null) {
+            fellBack.incrementAndGet();
+            lastFallbackReason = reason;
+        }
+    }
+
+    /** Lowers the traversal and returns null, or leaves it untouched and returns the reason. */
+    @SuppressWarnings({"rawtypes", "unchecked"})
+    private String plan(Traversal.Admin<?, ?> traversal) {
+        List<Step> steps = new ArrayList<>();
+        for (Step s : traversal.getSteps()) {
+            if (!(s instanceof NoOpBarrierStep)) {
+                steps.add(s);
+            }
+        }
+        if (steps.size() < 4) {
+            return "too few steps for V().out()..dedup().count()";
+        }
+        if (!(steps.get(0) instanceof GraphStep<?, ?> start) || !start.returnsVertex()) {
+            return "does not start with V()";
+        }
+        if (start instanceof HasContainerHolder holder && !holder.getHasContainers().isEmpty()) {
+            return "has() on the start step";
+        }
+        long[] seeds = seedRows(start.getIds());
+        if (seeds == null) {
+            return "V() needs explicit numeric row ids within the row store";
+        }
+        int last = steps.size() - 1;
+        if (!(steps.get(last) instanceof CountGlobalStep)) {
+            return "does not end with count()";
+        }
+        if (!(steps.get(last - 1) instanceof DedupGlobalStep<?> dedup)
+                || !dedup.getLocalChildren().isEmpty()
+                || !dedup.getScopeKeys().isEmpty()) {
+            return "count() without a plain dedup() counts walks, not vertices";
+        }
+        int[] hops = new int[last - 2];
+        for (int i = 1; i < last - 1; i++) {
+            // Provider strategies run in no fixed order, so has() can still be its own step
+            // here rather than folded into the start step.
+            if (steps.get(i) instanceof HasContainerHolder) {
+                return "has() at step " + i;
+            }
+            if (!(steps.get(i) instanceof VertexStep<?> v)) {
+                return "step " + i + " is not out(label): " + steps.get(i);
+            }
+            if (v.getDirection() != Direction.OUT || !v.returnsVertex()) {
+                return "step " + i + " is not an out() to vertices";
+            }
+            if (v.getEdgeLabels().length != 1) {
+                return "step " + i + " needs exactly one edge label";
+            }
+            Integer classid = edgeClassids.get(v.getEdgeLabels()[0]);
+            if (classid == null) {
+                return "edge label '" + v.getEdgeLabels()[0] + "' has no lance-graph classid";
+            }
+            hops[i - 1] = classid;
+        }
+        if (!steps.get(0).getLabels().isEmpty() || anyLabelled(steps)) {
+            return "as() labels would be lost";
+        }
+        TraversalHelper.removeAllSteps(traversal);
+        traversal.addStep(new LanceHopCountStep(traversal, store, seeds, hops));
+        lowered.incrementAndGet();
+        return null;
+    }
+
+    @SuppressWarnings("rawtypes")
+    private static boolean anyLabelled(List<Step> steps) {
+        for (Step s : steps) {
+            if (!s.getLabels().isEmpty()) {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    /** Row numbers for V(ids), or null when any id is not a row of this store. */
+    private long[] seedRows(Object[] ids) {
+        if (ids == null || ids.length == 0) {
+            return null;
+        }
+        long rows = store.rowCount();
+        long[] out = new long[ids.length];
+        for (int i = 0; i < ids.length; i++) {
+            Object id = ids[i] instanceof Vertex v ? v.id() : ids[i];
+            if (!(id instanceof Long || id instanceof Integer || id instanceof Short)) {
+                return null;
+            }
+            long row = ((Number) id).longValue();
+            if (row < 0 || row >= rows) {
+                return null;
+            }
+            out[i] = row;
+        }
+        return out;
+    }
+}