Skip to content

Commit 732b17d

Browse files
committed
Java surface is sql(), not the mask algebra; leak-on-throw fixes
Operator ruling: Java does not use masking ops — lance-graph does. Mask.minus() and RowStore.hop() are named as the wrong shape: the substrate's population algebra standing on the Java side of the wall. The Java surface is the boring call a developer already knows, handed down zero-copy, with the polyfill relation to sql() exactly a consumer crate's relation to ndarray::simd — the caller does not know why it is there, only that it is, and closing a gap is the Rust side's job. Consequences recorded: nobody optimizes Java except to make it more boring; a billion-row op crosses as a name and returns a scalar, so every public call is O(1) in rows and the one materialize* terminal is what makes "never" auditable; a mask concept on a public Java signature is a defect regardless of how well it works. The endgame that makes this non-negotiable rather than stylistic: low-code "bring your own software", where novel API is lock-in-by-learning-curve. Re-audited the materialization site list the rule depends on: it claimed five sites and the tree has seven. View.where()'s predicate copy and NativePattern.plan() were never listed. Both are legitimate and the reason sharpens the rule — they are bounded by the query the developer typed, not by the data, so the invariant is "nothing proportional to rows", not "no allocation". Also lands the four leak-on-throw fixes: Mask.ternlog, Mask.minus, RowStore.maskOfFacetTernaryMatch and RowStore.hop each moved their requireMinor ahead of allocation and now close the destination handle when the native op throws, via one shared package-private helper. No public signature changed. Gates: lgj-abi cargo test green, Java suite 409/409 against abi 0.11, ndarray::simd avx512, release. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
1 parent 079fcc7 commit 732b17d

7 files changed

Lines changed: 346 additions & 33 deletions

File tree

‎.claude/agents/jdk-toolchain-warden.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,20 @@ somewhere else**, and Java is never told. That fluency is exactly what a
8585
menu" and "a pleasing menu that lies about what the kitchen did." Shipping it
8686
unverified forfeits the product, not a test.
8787

88+
> **⊘ CORRECTED, same day — the menu is BORING, and the example above is the
89+
> wrong one.** Operator: *"Java doesnt use masking ops. `Mask.minus()`,
90+
> `RowStore.hop()`. Lance-graph does. Java just sees boring `sql()` handed to
91+
> duckdb."* and *"nobody should ever start trying to optimize Java (except
92+
> making it boring front)."* So `view.where(..).hop(..).count()` is the mask
93+
> algebra on the wrong side of the wall, not the product. Read the paragraph
94+
> above with `sql("select …")` in its place; every word of it still holds, and
95+
> the reason the 409-check run matters is unchanged — it is what keeps a
96+
> boring surface honest instead of a disguise, since **the boringness is handed
97+
> down zero-copy**. The endgame: low-code *"Bring your own software"* against
98+
> Palantir Foundry, where novel API is lock-in-by-learning-curve and therefore
99+
> the enemy. Full ruling: `CLAUDE.md` § "THE JAVA SURFACE IS `sql()`, NOT THE
100+
> MASK ALGEBRA".
101+
88102
The allocation the menu sits on (operator, same day):
89103

90104
| what | lives in | membrane that keeps Java out of it |

‎CLAUDE.md‎

Lines changed: 162 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -230,9 +230,24 @@ build):
230230
not row count — verified fixed-size on both the Rust and Java sides);
231231
`Abi.java`'s `readCarvings` (bounded by `CARVING_SLOTS`, a manifest
232232
constant, not n_rows); `Engine.facetSumResolved`'s fixed `long[2]`
233-
result pair. None of the five is a hidden proportional-to-n_rows
234-
population copy — keep this list exhaustive when a sixth site is added,
235-
rather than letting the enumeration silently go stale again.
233+
result pair; `View.where()`'s `List.copyOf` of the PREDICATE chain; and
234+
`NativePattern.plan()`'s `predicates.stream()…toList()`. None of the
235+
seven is a hidden proportional-to-n_rows population copy — keep this list
236+
exhaustive when an eighth site is added, rather than letting the
237+
enumeration silently go stale again.
238+
239+
> **⊘ RE-AUDITED 2026-09-16 and the list WAS stale — it claimed five and
240+
> the tree had seven.** `View.where()` and `NativePattern.plan()` were
241+
> never listed. Both are legitimate, and *why* is the sharper statement of
242+
> the rule: they are bounded by the number of predicates the developer
243+
> CHAINED, i.e. by the size of the query they typed — never by the data.
244+
> So the invariant is not "no allocation on the Java side"; it is
245+
> **nothing proportional to ROWS**, and a query-shaped allocation is the
246+
> boring front's own size, not the substrate's. The audit that found this
247+
> is one grep (`long[]`, `toArray`, `copyOf`, `.stream()` across
248+
> `java/src/main`) and it is the check to re-run before citing the list —
249+
> the enumeration had already gone stale once under a sentence telling the
250+
> next session not to let it.
236251
Temporary kernel scratch (SIMD scratch masks, decode buffers) is
237252
allowed and is NOT the same claim as a second canonical copy.
238253
- **Layout parity is independently derived, not self-compared.**
@@ -328,6 +343,150 @@ simd_{amx,avx512,avx2, Rust: lgj-abi kernels → ndarray::simd
328343
> holds at the TOP row (facade ↔ facade) and the BOTTOM row (backends ↔ the
329344
> Rust floor). It BREAKS in the middle. Do not reason from the middle row.
330345
346+
## THE JAVA SURFACE IS `sql()`, NOT THE MASK ALGEBRA (operator-ruled, 2026-09-16)
347+
348+
**Verbatim, in four parts:**
349+
350+
> *"Java doesnt use masking ops. `Mask.minus()`, `RowStore.hop()`. Lance-graph
351+
> does. Java just sees boring `sql()` handed to duckdb (Example)."*
352+
>
353+
> *"nobody should ever start trying to optimize Java (except making it boring
354+
> front)."*
355+
>
356+
> *"the boringness is then handed to lancegraph as zero copy."*
357+
>
358+
> *"and handled akin to ndarray polyfill — java doesnt know why there is
359+
> `sql()` polyfill, we just make sure there is."*
360+
361+
### What this corrects, by name
362+
363+
⊘ The sentence three paragraphs up — *"offers the menu to the table in a
364+
pleasing way, **using masking ops**"* — is **struck on those three words**.
365+
The menu framing survives; the mechanism named in it does not. lance-graph
366+
uses masking ops. Java does not, must not, and is never told they exist.
367+
368+
⊘ The isomorphism table's top row previously read `Java (View / Mask /
369+
RowStore / consumers)` as the facade. **`Mask` and `RowStore` are NOT the
370+
facade** — they are the substrate's algebra standing on the Java side of the
371+
wall. `Mask.minus()` and `RowStore.hop()` are the operator's own two examples
372+
of the wrong shape. The facade is the boring call a Java developer already
373+
knows how to write.
374+
375+
### The polyfill relation is EXACT, and it is the whole design
376+
377+
This is the part that makes the ruling operational rather than stylistic.
378+
`ndarray::simd` is a facade with **37 functions and zero shipping
379+
instructions** — a consumer crate calls `U8x64::cmpeq_mask` and has no idea
380+
whether AVX-512, NEON, wasm or scalar answered it. The consumer does not
381+
know why the polyfill exists. It only knows it is there.
382+
383+
`sql()` is that, one tier up:
384+
385+
| tier | the caller writes | the caller does not know |
386+
|---|---|---|
387+
| consumer crate → ndarray | `U8x64::cmpeq_mask(..)` | which of six backends ran |
388+
| Java → lance-graph | `sql("select …")` | that masks, ternlog, hops or popcounts exist at all |
389+
390+
**"We just make sure there is one"** is the standing obligation, and it falls
391+
on the Rust side, never on Java. A missing `sql()` capability is a
392+
lance-graph/ABI gap to close — exactly as a missing SIMD primitive is an
393+
`ndarray::simd` gap to close, never a licence for a consumer to write
394+
intrinsics. This is the MISSING-CAPABILITY STOP RULE already in this file,
395+
now with its Java-side spelling.
396+
397+
### The three operational consequences
398+
399+
1. **Nobody optimizes Java. Ever.** The only sanctioned work on the Java side
400+
is making it **more boring** — more ordinary, more familiar, closer to what
401+
a Java developer would have written without us. A PR whose stated goal is a
402+
faster Java path is rejected on its goal, before its diff is read. "Boring"
403+
is the performance strategy: the speed is elsewhere.
404+
2. **The boringness is handed down as zero copy.** The ordinary-looking call
405+
is not translated, re-encoded, or marshalled. It crosses as a NAME and the
406+
bytes stay put — which is what makes the boring surface honest rather than
407+
a disguise. Zero-copy is not an optimization applied to the glove; it is
408+
the condition under which a boring glove is allowed to exist.
409+
3. **A mask concept on a public Java signature is a defect**, regardless of
410+
how well it works. `java-surface-warden` already blocks byte positions and
411+
arithmetic; this ruling adds the population algebra by name — `Mask`,
412+
`minus`, `hop`, `ternlog`, `popcount`, lane ids, opcodes. Those are
413+
T1/T2 words (see `kernel-membrane-warden`), and T3 does not speak them.
414+
415+
4. **Java never does materialization — and the row count is what proves it**
416+
(operator, same day): *"Java never does materialization. A billion rows op
417+
is handed behind the front and handled as a masking hattrick nobody knew
418+
what's coming."* This is the boring front's load-bearing property, not a
419+
performance note. A billion-row operation crosses as a NAME and returns a
420+
scalar; the billion never exists on the Java side, in any form, at any
421+
moment. That is why the surface can afford to be boring: `sql()` looks
422+
identical at 10 rows and at 10^9, because Java's work is the same at both —
423+
none.
424+
425+
**The checkable form of "never":** every public Java call must be
426+
**O(1) in rows**, and the ONLY exit is a method whose name begins with
427+
`materialize` (today: exactly one, `Mask.materializeRows()`, O(n), stated
428+
in its javadoc). The naming rule is not a loophole in "never" — it is the
429+
mechanism that makes "never" auditable, because anything proportional to
430+
row count that is NOT named that way is a defect by inspection, with no
431+
judgement call required. Keep the exhaustive materialization-site list in
432+
the zero-copy section current for exactly this reason: an unnamed sixth
433+
site is the failure this rule exists to catch.
434+
435+
**The corollary that bites hardest in review:** a Java-side loop is not
436+
merely slow, it is *proof that something was materialized to loop over*.
437+
E1 forbids the loop; this forbids the thing that made a loop expressible.
438+
Two statements of one rule from opposite ends.
439+
440+
### THE ENDGAME — why "boring" is strategy, not taste (operator, same day)
441+
442+
> *"endgame is make Java low code **'Bring your own software'** to beat
443+
> palantir foundry at its own game."*
444+
445+
Foundry's proposition is *bring your data to our platform* — and then learn
446+
its ontology tooling, its pipeline builder, its workshop, its idioms. The
447+
learning curve is not a cost of the product; **it IS the product**, because
448+
every hour a customer spends learning it is an hour of lock-in that a
449+
competitor must refund before they can switch.
450+
451+
BYOS inverts exactly that: **bring your own software.** The customer keeps
452+
their Java, their SQL, their JDBC-shaped habits, their existing build — and
453+
the substrate is underneath without them learning a new vocabulary to reach
454+
it. Nothing to port, nothing to adopt, nothing to unlearn if they leave.
455+
456+
**This is what makes each of the three consequences above non-negotiable
457+
rather than stylistic:**
458+
459+
- **Novel API is the enemy, not slow API.** `Mask.minus()` is something a
460+
customer must LEARN; `sql()` is something they already know. Every unit of
461+
novelty on the Java surface is a unit of Foundry-shaped lock-in-by-curve —
462+
built by us, against our own pitch. That is why optimizing Java loses the
463+
game even when it succeeds: optimization on that side produces novel API,
464+
and novel API is precisely the thing BYOS promises not to require.
465+
- **"Boring" is therefore the competitive moat, measured as absence.** The
466+
win condition is that a Java developer writes what they would have written
467+
anyway and the result arrives at substrate speed. There is no demo of this
468+
that looks impressive — it looks like nothing happened, which is the point.
469+
- **The advantage must live where the customer does not have to look.** Speed
470+
comes from lance-graph and ndarray; the customer gets it by writing ordinary
471+
code. A surface that has to be studied to be fast has already conceded the
472+
argument, however fast it then is.
473+
474+
**Practical test for any proposed Java-side addition:** *would a customer who
475+
has never read our documentation write this line by accident, from habit
476+
alone?* If yes, it is a candidate. If it needs a paragraph of explanation, it
477+
is Foundry's business model with our name on it — reject it and close the gap
478+
on the Rust side instead.
479+
480+
### Scope — what this does NOT do
481+
482+
It does not delete `Mask` or `RowStore` today. They exist, they are tested,
483+
and the four leak-on-throw fixes that landed with this ruling make existing
484+
code correct rather than expanding it. What the ruling settles is the
485+
DIRECTION: the boring `sql()`-shaped surface is the target, the mask algebra
486+
is not to be grown on the Java side, and no future session may cite the
487+
struck sentence above as licence to add another `Mask.*` verb.
488+
489+
331490
Measured grounding (2026-08-27, in-tree): `simd.rs` is 37 functions and
332491
ZERO shipping instructions — every raw intrinsic in it sits inside
333492
`#[cfg(test)]` as the wrapper's oracle; `simd_avx512.rs` alone carries 488.

‎java/src/main/java/com/adaworldapi/lancegraph/Mask.java‎

Lines changed: 56 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package com.adaworldapi.lancegraph;
22

3+
import com.adaworldapi.lancegraph.internal.ffm.Abi;
34
import com.adaworldapi.lancegraph.internal.ffm.Engine;
45

56
/**
@@ -90,8 +91,22 @@ public Mask minus(Mask other) {
9091
java.util.Objects.requireNonNull(other, "other");
9192
requireUsable("minus()");
9293
other.requireUsable("minus()'s argument");
94+
// Same reasoning as ternlog() below: a pure manifest check, no handle touched, run
95+
// BEFORE dst is allocated rather than left solely to Engine.maskAndNot's own copy of the
96+
// same guard, so a too-old library never leaves an orphaned mask behind.
97+
Abi.requireMinor(4);
9398
long dst = Engine.createMask(resourceHandleOf(parent), false);
94-
Engine.maskAndNot(handle, other.handle, dst);
99+
try {
100+
Engine.maskAndNot(handle, other.handle, dst);
101+
} catch (LanceGraphException e) {
102+
// The version gate above already passed; a failure here is the actual op
103+
// (MASK_LENGTH_MISMATCH if other belongs to a mismatched population). dst was
104+
// allocated for a Mask this method never got to construct, so nothing else owns it
105+
// -- release it before the failure propagates, or it and its registry slot leak for
106+
// the life of the process.
107+
closeOnFailure(dst, e);
108+
throw e;
109+
}
95110
return new Mask(parent, dst);
96111
}
97112

@@ -134,8 +149,23 @@ public Mask ternlog(Mask b, Mask c, int imm) {
134149
requireUsable("ternlog()");
135150
b.requireUsable("ternlog()'s b argument");
136151
c.requireUsable("ternlog()'s c argument");
152+
// A pure manifest check, no handle touched -- run BEFORE dst is allocated rather than
153+
// left solely to Engine.maskTernlog's own copy of the same guard, so a too-old library
154+
// never leaves an orphaned mask behind (there is nothing yet to leak). Engine.maskTernlog
155+
// still carries its own requireMinor(11) too, for any caller that reaches it directly.
156+
Abi.requireMinor(11);
137157
long dst = Engine.createMask(resourceHandleOf(parent), false);
138-
Engine.maskTernlog(handle, b.handle, c.handle, dst, (byte) imm);
158+
try {
159+
Engine.maskTernlog(handle, b.handle, c.handle, dst, (byte) imm);
160+
} catch (LanceGraphException e) {
161+
// The version gate above already passed; a failure here is the actual op (e.g.
162+
// MASK_LENGTH_MISMATCH if b/c belong to a mismatched population). dst was allocated
163+
// for a Mask this method never got to construct, so nothing else owns it -- release
164+
// it before the failure propagates, or it and its registry slot leak for the life of
165+
// the process.
166+
closeOnFailure(dst, e);
167+
throw e;
168+
}
139169
return new Mask(parent, dst);
140170
}
141171

@@ -329,6 +359,30 @@ private static long resourceHandleOf(NativeResource resource) {
329359
+ " implementation: " + resource.getClass());
330360
}
331361

362+
/**
363+
* Release a mask allocated as a would-be result, after the fallible native call meant to
364+
* populate it failed before a {@link Mask} could be constructed to own it — the shared
365+
* close-on-failure step for every {@code Engine.createMask(...)} + fallible-op pair on this
366+
* facade ({@link #ternlog}; {@link RowStore#maskOfFacetTernaryMatch} reaches this too, since
367+
* it is package-private "for peers", exactly like {@link #handle()} just above per the
368+
* section comment introducing it).
369+
*
370+
* <p>Without this, {@code handleToClose}'s native allocation and registry slot would leak
371+
* silently: there is no {@link Mask} object anywhere for a caller to close, because the
372+
* constructor that would have handed one out never ran. The original failure is always what
373+
* propagates to the caller — a failure while releasing {@code handleToClose} is attached to
374+
* it as a {@linkplain Throwable#addSuppressed suppressed} exception rather than replacing it,
375+
* so diagnosing "why did the operation fail" is never hijacked by "why did the cleanup of the
376+
* failed operation fail".
377+
*/
378+
static void closeOnFailure(long handleToClose, LanceGraphException primary) {
379+
try {
380+
Engine.close(handleToClose);
381+
} catch (LanceGraphException cleanupFailure) {
382+
primary.addSuppressed(cleanupFailure);
383+
}
384+
}
385+
332386
private void requireUsable(String what) {
333387
if (closed) {
334388
throw new ClosedResourceException(what + " was called on a closed selection");

‎java/src/main/java/com/adaworldapi/lancegraph/RowStore.java‎

Lines changed: 36 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package com.adaworldapi.lancegraph;
22

3+
import com.adaworldapi.lancegraph.internal.ffm.Abi;
34
import com.adaworldapi.lancegraph.internal.ffm.Engine;
45
import com.adaworldapi.lancegraph.internal.ffm.Layouts;
56

@@ -192,9 +193,28 @@ public Mask maskOfFacetTernaryMatch(FacetId facet, long patternLo64, int pattern
192193
long careLo64, int careHi32) {
193194
java.util.Objects.requireNonNull(facet, "facet");
194195
requireOpen("maskOfFacetTernaryMatch()");
196+
// A pure manifest check, no handle touched -- run BEFORE mask is allocated rather than
197+
// left solely to Engine.ternaryMatch's own copy of the same guard, so a too-old library
198+
// never leaves an orphaned mask behind (there is nothing yet to leak). Engine.ternaryMatch
199+
// still carries its own requireMinor(11) too, for any caller that reaches it directly.
200+
// Mirrors Mask.ternlog's own reasoning (see that method).
201+
Abi.requireMinor(11);
195202
long mask = Engine.createMask(handle, false);
196-
Engine.ternaryMatch(handle, facet.index(), patternLo64, patternHi32, careLo64, careHi32,
197-
mask);
203+
try {
204+
Engine.ternaryMatch(handle, facet.index(), patternLo64, patternHi32, careLo64,
205+
careHi32, mask);
206+
} catch (LanceGraphException e) {
207+
// The version gate above already passed; a failure here is the actual op --
208+
// UNSUPPORTED_LAYOUT on a columnar store (see this method's own javadoc), or a
209+
// MASK_LENGTH_MISMATCH. mask was allocated for a Mask this method never got to
210+
// construct, so nothing else owns it -- release it before the failure propagates, or
211+
// it and its registry slot leak for the life of the process. Mask.closeOnFailure is
212+
// package-private "for peers" (the same standing Mask.handle() already grants
213+
// RowStore) precisely so this site does not need to reinvent the same
214+
// close-then-suppress shape.
215+
Mask.closeOnFailure(mask, e);
216+
throw e;
217+
}
198218
return new Mask(this, mask);
199219
}
200220

@@ -344,8 +364,21 @@ public Mask hop(int edgeClassid, WideFieldMask facets, Mask src) {
344364
java.util.Objects.requireNonNull(facets, "facets");
345365
java.util.Objects.requireNonNull(src, "src");
346366
requireOpen("hop()");
367+
// Same reasoning as maskOfFacetTernaryMatch() above: a pure manifest check, no handle
368+
// touched, run BEFORE dst is allocated rather than left solely to Engine.hop's own copy
369+
// of the same guard, so a too-old library never leaves an orphaned mask behind.
370+
Abi.requireMinor(4);
347371
long dst = Engine.createMask(handle, false);
348-
Engine.hop(handle, edgeClassid, facets.bits(), 0, src.handle(), dst);
372+
try {
373+
Engine.hop(handle, edgeClassid, facets.bits(), 0, src.handle(), dst);
374+
} catch (LanceGraphException e) {
375+
// The version gate above already passed; a failure here is the actual op. dst was
376+
// allocated for a Mask this method never got to construct, so nothing else owns it
377+
// -- release it before the failure propagates, or it and its registry slot leak for
378+
// the life of the process. Reuses Mask.closeOnFailure rather than a second helper.
379+
Mask.closeOnFailure(dst, e);
380+
throw e;
381+
}
349382
return new Mask(this, dst);
350383
}
351384

0 commit comments

Comments
 (0)