Skip to content

Commit 1c66091

Browse files
committed
Panama x Valhalla integration: falsifier, full build surface, lab re-scoped, measurements
Completes the integration rather than the toolchain bump. - RangeOutOfBounds: compatibility plumbing only, now backed by plan_lower::range_falsifier -- sweeps every opcode through the real lower_plan, disable-verified red-then-green. Goes red the day LGJ gains a Range lowering, forcing a deliberate caller-visible mapping. - Build surface: java/README.md, README.md, docs/panama.md, bench/gate-run.sh and two agent cards retargeted to JDK 28, each stating that the preview flag is Valhalla's and never FFM's. Historical result artifacts left untouched. - valhalla-lab re-scoped: both arms on one JDK, so the object model is the only variable (it was confounded with the JDK version before). Adapted the flattening probe to JDK 28's narrowed ValueClass.isFlatArray signature. - Measured: 183 native tests, 409 Java checks; LaneId/Ordinal/MaskId flatten, RowRange and Row do not (the 8-byte cliff reproduces); array construction 8.2x less allocation, non-escaping construction and Row hydration slightly worse; the FFM path is byte-identical across the object model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HScwwezRdMxFfTs3WLG19d
1 parent 6ca9c29 commit 1c66091

22 files changed

Lines changed: 476 additions & 271 deletions

‎.claude/agents/BOOT.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,11 @@ matched to the actual size of the problem.
2424
`jextract`/`cbindgen` out of habit).
2525
5. **`.claude/knowledge/jdk-toolchain-facts.md`** — which JDK path to
2626
use for which purpose. Getting this wrong (e.g. using `/usr/bin/java`
27-
instead of `/opt/jdks/jdk-26.0.2`) produces confusing preview-flag
27+
instead of `/opt/jdks/jdk-28+16`) produces confusing preview-flag
2828
errors that look like a design problem but are a toolchain-selection
29-
mistake.
29+
mistake. On JDK 28 the production build carries `--release 28
30+
--enable-preview` for the Valhalla value types — FFM itself needs no
31+
preview flag.
3032
6. **`.claude/knowledge/agent-cargo-hygiene.md`** — operator directive:
3133
spawned agents do NOT run `cargo` in any form (build/check/test/
3234
clippy), ever. Only the orchestrating main thread compiles. This

‎.claude/agents/panama-bridge-engineer.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,9 +63,12 @@ on both sides.
6363
Java side actually implements its half.
6464
7. **`--enable-native-access` is required and documented** in
6565
`java/README.md`'s exact command lines — verify the commands there
66-
actually run against `/opt/jdks/jdk-26.0.2` (see
67-
`.claude/knowledge/jdk-toolchain-facts.md`) without additional flags
68-
beyond that one.
66+
actually run against `/opt/jdks/jdk-28+16` (see
67+
`.claude/knowledge/jdk-toolchain-facts.md`). **Two flags, two owners:**
68+
`--enable-native-access=ALL-UNNAMED` is FFM's and the only one Panama
69+
needs; `--enable-preview` is VALHALLA's, required because the production
70+
vocabulary types are `value record`s. Never attribute the preview flag to
71+
FFM — on JDK 28 FFM is final.
6972

7073
## What you are not
7174

‎.claude/board/EPIPHANIES.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,107 @@
1+
## E-PANAMA-TIMES-VALHALLA-IS-ONE-MEMBRANE-AND-THE-LAB-FINALLY-HAS-ONE-VARIABLE-1 (2026-09-19)
2+
3+
**Operator reframing, and it is the correct one:** this was never a JDK 26→28
4+
migration. It is **Panama-only production + Valhalla-lab arm → one production
5+
Panama × Valhalla membrane.** JDK 28 is the enabling toolchain, not the point.
6+
Panama answers *where are the bytes and how does Java reach them without
7+
copying*; Valhalla answers *how does Java name and carry semantic values over
8+
those bytes without identity-heavy heap objects*. lance-graph / T0 remains the
9+
only owner of canonical storage.
10+
11+
### Measured on JDK 28 (`/opt/jdks/jdk-28+16`), production value-shaped
12+
13+
| gate | result |
14+
|---|---|
15+
| native `lgj-abi` | **183 tests** green (182 + the new Range falsifier) |
16+
| Java `AllTests`, production as `value record` | **ALL PASSED (409 checks)** — identical to the unflipped baseline on the same JDK |
17+
| all six vocabulary types | `isValue() == true`; substitutability holds |
18+
19+
### The flattening cliff is UNCHANGED on JDK 28 — and it bites one of our six
20+
21+
Measured by the lab, both arms on one JDK:
22+
23+
| type | payload | array |
24+
|---|---|---|
25+
| `LaneId` (1 int) | 4 B | **FLAT** |
26+
| `Ordinal` (1 int) | 4 B | **FLAT** |
27+
| `MaskId` (1 long) | 8 B | **FLAT** |
28+
| `RowRange` (2 long) | 16 B | **NOT-FLAT** |
29+
| `Row` (1 long + 2 int) | 16 B | **NOT-FLAT** |
30+
31+
The ≤8-byte cliff measured on the 27-jep401ea3 build **reproduces on JDK 28**.
32+
So `value semantics` and `flattening` remain two different claims, and
33+
**`RowRange` is a real production type on the wrong side of the cliff.** No
34+
performance claim may be made for it from `isValue() == true`.
35+
36+
### Allocation deltas, record arm → value arm (1M ops, same JDK)
37+
38+
| measurement | record | value | delta |
39+
|---|---|---|---|
40+
| construct N `LaneId` into an array | 15.26 MiB | **1.87 MiB** | **8.2× less** (16 B → 1.96 B each) |
41+
| allocate+fill `LaneId[N]` | 19.07 MiB | **5.65 MiB** | 3.4× less |
42+
| construct N `Descriptor` (2 wrappers) | 45.78 MiB | **27.89 MiB** | 1.6× less |
43+
| construct N `LaneId`, never escaping | 4.73 MiB | **5.75 MiB** | ⊘ **value arm allocates MORE** — escape analysis already erased the record case |
44+
| hydrate 65,536 `Row` | 1.50 MiB | **2.00 MiB** | ⊘ **worse** — `Row` is 16 B, NOT-FLAT |
45+
46+
**Two of the five went the wrong way, and both are honest.** Where EA already
47+
won, value classes add nothing; where the shape does not flatten, hydration
48+
costs more. Production does not hydrate rows, so the second is a warning about
49+
a path we do not take — not a regression we shipped.
50+
51+
### The Panama path is byte-identical across the object model
52+
53+
| FFM measurement | record arm | value arm |
54+
|---|---|---|
55+
| Java bytes allocated per query (warm) | 712 B | **712 B** |
56+
| Java objects per row | 0 | **0** |
57+
| native lane bytes / mask bytes | 1.00 MiB / 8.0 KiB | **identical** |
58+
59+
**That is the integration's central invariant, measured rather than asserted:**
60+
Valhalla changes what the *vocabulary* costs; it does not touch the membrane.
61+
No second graph representation appeared, and no row hydration was introduced
62+
to "use Valhalla".
63+
64+
### The lab's re-scope removed a confound it had carried since it was built
65+
66+
The old A/B was `(record, JDK 26)` vs `(value record, JDK 27 EA)` — **two
67+
variables**. Both arms now run on JDK 28, so the object model is the only
68+
difference and the diff is attributable. The lab stops being
69+
present-vs-future and becomes a VM/representation probe suite over the real
70+
production vocabulary.
71+
72+
Two things the re-scope forced, both real:
73+
- **Preview marking is transitive.** Production classfiles are preview-marked,
74+
so the `record` arm *also* runs `--enable-preview`. It is a record arm on a
75+
preview-enabled JVM, not a preview-free arm; the "stable" label survives only
76+
as a path.
77+
- **A JDK 28 internal-API change.** `jdk.internal.value.ValueClass.isFlatArray`
78+
narrowed from `Object` to `Object[]` since 27-jep401ea3; the flattening probe
79+
was adapted (a non-`Object[]` argument answers `NOT-FLAT` without reaching
80+
the internal API).
81+
82+
### The `RangeOutOfBounds` fix is compatibility plumbing, with a falsifier
83+
84+
`mask_risc::ExecError` gained `RangeOutOfBounds` with lance-graph's
85+
`Pred::Range` work and `exec_error_to_status` matches exhaustively, so
86+
`lgj-abi` **did not compile against lance-graph `main`**. Mapped into the
87+
documented *"bug in THIS file"* family — no new public status, no ABI bump —
88+
and backed by `plan_lower::range_falsifier::no_opcode_lowers_to_pred_range`,
89+
which sweeps every opcode through the real `lower_plan` and reads every
90+
emitted predicate. **Disable-verified red-then-green:** pointing `LGJ_OP_EQ_U32`
91+
at `Pred::Range` fails it with the intended message; restoring passes. When LGJ
92+
gains a Range lowering this goes red and forces a deliberate caller-visible
93+
mapping instead of inheriting `LGJ_ERR_ALLOCATION_FAILED`, which would then be
94+
a lie.
95+
96+
### What could not be measured here, stated rather than implied
97+
98+
`bench/` cannot run in this container — `bench/lib` does not exist, so the JMH
99+
jars are absent. The crossing-count and per-call allocation figures in
100+
`bench/RESULTS.md` are **historical JDK 26 measurements, correct when taken**,
101+
and are deliberately left untouched; they have NOT been re-taken on JDK 28.
102+
`gate-run.sh` carries the new pin and flags so the re-run is one command once
103+
the jars are present.
104+
1105
## E-THE-VALHALLA-FLIP-COST-ONE-WORD-SIX-TIMES-AND-THE-GATES-DID-NOT-MOVE-1 (2026-09-19)
2106

3107
**Finding, measured end to end today.** The P0 mandate (JDK 28 + Valhalla +

‎README.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -70,11 +70,13 @@ no `cbindgen`, no `jextract`, no JNI. See `docs/abi.md` §1 and
7070
## Toolchains
7171

7272
- Rust: **1.97.1** stable, matching `ndarray`/`lance-graph`'s pin.
73-
- Production JDK: a GA build with `java.lang.foreign` final (no preview
74-
flag needed) — verified against JDK 26 GA.
75-
- Valhalla lab JDK: the official JEP 401 early-access binary from
76-
`jdk.java.net/valhalla/` — a value-classes preview build, kept physically
77-
separate from the production path.
73+
- Production JDK: **JDK 28** (`/opt/jdks/jdk-28+16`). Panama FFM is final
74+
there and needs no flag; Valhalla (JEP 401) is preview-gated and production
75+
uses it, so `--enable-preview` is required — for the value classes, never
76+
for FFM.
77+
- There is no separate "Valhalla JDK" any more. Production **is** the Valhalla
78+
arm; `valhalla-lab/` is now a VM/representation probe suite over the real
79+
production vocabulary rather than a parallel future-vs-present A/B.
7880

7981
See `.claude/knowledge/jdk-toolchain-facts.md` for exact verified paths and
8082
flags in the reference development environment.

‎bench/gate-run.sh‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ set -euo pipefail
1010

1111
BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
1212
ROOT="$(cd "$BENCH/.." && pwd)"
13-
JDK="${JDK:-/opt/jdks/jdk-26.0.2}"
13+
JDK="${JDK:-/opt/jdks/jdk-28+16}" # P0: JDK 28 + Valhalla (JEP 401, preview) + Panama FFM
1414
LIB_DIR="${LIB_DIR:-$ROOT/target/release}"
1515
OUT="$BENCH/out-gate"
1616
CP_LIBS="$(find "$BENCH/lib" -name '*.jar' | sort | tr '\n' ':')"
@@ -27,10 +27,10 @@ cmp -s "$PROBE" "$VARIANT" && { echo "FAIL: variant is identical to the shipped
2727
rm -rf "$OUT"; mkdir -p "$OUT/before" "$OUT/after" "$OUT/bench" "$BENCH/results"
2828

2929
echo "== build BEFORE: java/src/main as shipped"
30-
"$JDK/bin/javac" -d "$OUT/before" $(find "$API" -name '*.java')
30+
"$JDK/bin/javac" --release 28 --enable-preview -d "$OUT/before" $(find "$API" -name '*.java')
3131

3232
echo "== build AFTER: same tree, LaneProbe.java swapped for bench/variants/probed"
33-
"$JDK/bin/javac" -d "$OUT/after" $(find "$API" -name '*.java' ! -path "$PROBE") "$VARIANT"
33+
"$JDK/bin/javac" --release 28 --enable-preview -d "$OUT/after" $(find "$API" -name '*.java' ! -path "$PROBE") "$VARIANT"
3434

3535
# One compiled bench, run against each API build (the bench class is identical by construction).
3636
echo "== build the bench once"
@@ -41,6 +41,7 @@ run_arm() {
4141
local arm="$1"
4242
echo "== run $arm"
4343
( cd "$BENCH" && JAVA_TOOL_OPTIONS= "$JDK/bin/java" \
44+
--enable-preview \
4445
--enable-native-access=ALL-UNNAMED --add-modules jdk.incubator.vector \
4546
-Dstdout.encoding=UTF-8 -Dlgj.library="$LIB_DIR/liblgj_abi.so" \
4647
-Dlgj.bench.result="results/gate-$arm.csv" \

‎docs/panama.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -70,11 +70,17 @@ correctly (`bench/README.md` rule 4). Measured: a bare downcall costs
7070
~22 ns over a plain Java call (`bench/RESULTS.md`, Component A) — the
7171
floor every native-side operation in this project sits on top of.
7272

73-
## `--enable-preview` never reaches the shipped path
74-
75-
The production Java tree (`java/`) targets `/opt/jdks/jdk-26.0.2`, where
76-
FFM is **final** — the only flag needed anywhere is
77-
`--enable-native-access=ALL-UNNAMED`. The Valhalla lab
73+
## `--enable-preview` is Valhalla's flag, never FFM's
74+
75+
⊘ This section formerly read *"`--enable-preview` never reaches the shipped
76+
path"*, which was true only while production and Valhalla were split arms.
77+
The production Java tree (`java/`) now targets `/opt/jdks/jdk-28+16`, where
78+
**FFM is final and still needs no flag** — `--enable-native-access=ALL-UNNAMED`
79+
remains the only thing Panama asks for. What changed is that production
80+
deliberately uses Valhalla value classes (JEP 401, preview on this build), so
81+
`--release 28 --enable-preview` is now part of the shipped build contract.
82+
Read every "no preview" claim below as scoped to FFM: **Panama needs no
83+
preview; the vocabulary types do.** The Valhalla lab
7884
(`valhalla-lab/src/valhalla`) is compiled *separately*, with its own
7985
`-source 27 --enable-preview`, into its own output directory
8086
(`results/valhalla-lab/`), and is never on the classpath the production

‎java/README.md‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,17 +27,20 @@ become one native lane set, one packed mask, a few tiny schema descriptors, and
2727
There is no Maven, no Gradle, no downloaded dependency, and no C toolchain. `javac` and `java` are
2828
the entire Java toolchain, exactly as `cargo` is the entire Rust one.
2929

30-
- **JDK 26** (`/opt/jdks/jdk-26.0.2`). FFM is **final** in JDK 26, so `--enable-preview` is neither
31-
needed nor accepted. Do **not** use a JDK 21 `java` on the path — there FFM is preview-gated and
32-
these commands will not work.
30+
- **JDK 28** (`/opt/jdks/jdk-28+16`). Two features, two reasons: **Panama FFM is final** here, so
31+
it needs no flag; **Valhalla (JEP 401) is preview-gated**, and production deliberately uses it —
32+
the six vocabulary types (`LaneId`, `MaskId`, `Ordinal`, `FacetId`, `RowRange`, `WideFieldMask`)
33+
are `value record`s. So `--release 28 --enable-preview` is required to COMPILE and `--enable-preview`
34+
to RUN. The flag is Valhalla's, never FFM's. Do **not** use a JDK 21 `java` on the path — there FFM
35+
is preview-gated too and these commands will not work.
3336
- No JNI, no `jextract`, no `cbindgen`, no `.h` file anywhere in the project. See `../docs/abi.md`
3437
§0 for why those are absent by construction rather than by preference.
3538

3639
## Build
3740

3841
```sh
3942
cd java
40-
/opt/jdks/jdk-26.0.2/bin/javac -d out $(find src/main/java src/test/java -name '*.java')
43+
/opt/jdks/jdk-28+16/bin/javac --release 28 --enable-preview -d out $(find src/main/java src/test/java -name '*.java')
4144
```
4245

4346
Compilation emits **seven** `[restricted]` warnings with `-Xlint:all`: six in
@@ -56,11 +59,11 @@ exercise the restriction itself.
5659

5760
```sh
5861
# everything
59-
/opt/jdks/jdk-26.0.2/bin/java --enable-native-access=ALL-UNNAMED -cp out \
62+
/opt/jdks/jdk-28+16/bin/java --enable-preview --enable-native-access=ALL-UNNAMED -cp out \
6063
com.adaworldapi.lancegraph.AllTests
6164

6265
# or one suite at a time — each has its own main
63-
/opt/jdks/jdk-26.0.2/bin/java --enable-native-access=ALL-UNNAMED -cp out \
66+
/opt/jdks/jdk-28+16/bin/java --enable-preview --enable-native-access=ALL-UNNAMED -cp out \
6467
com.adaworldapi.lancegraph.SmokeTest
6568
```
6669

@@ -69,7 +72,7 @@ exercise the restriction itself.
6972
| Flag | Needed? | Why |
7073
|---|---|---|
7174
| `--enable-native-access=ALL-UNNAMED` | **yes** | FFM's restricted methods (`libraryLookup`, `downcallHandle`, `reinterpret`) refuse to run without it. Omitting it does not fail the build — it fails at load, which is worse. |
72-
| `--enable-preview` | **no** | FFM is final in JDK 26. Passing it is an error. |
75+
| `--enable-preview` | **yes** | NOT for FFM — FFM is final on JDK 28. It is required because the production vocabulary types are Valhalla `value record`s (JEP 401, preview on this build). |
7376
| `-Djava.library.path=…` | **no** | Not used. That is the JNI mechanism; this project resolves the artifact itself (below). |
7477

7578
### Where the native library is found

‎native/lgj-abi/src/plan_lower.rs‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,3 +158,62 @@ fn pred_of(op: &LgjOpDesc) -> Option<Pred> {
158158
_ => return None,
159159
})
160160
}
161+
162+
#[cfg(test)]
163+
mod range_falsifier {
164+
use super::*;
165+
use crate::abi::LGJ_COMBINE_OR;
166+
167+
/// **The falsifier behind `exec_error_to_status`'s `RangeOutOfBounds` arm.**
168+
///
169+
/// That arm maps a `mask_risc` error into the documented *"would be a bug
170+
/// in THIS file, not in a caller's plan"* family, and the justification is
171+
/// one sentence: **this lowering emits no `Pred::Range`**, exactly as it
172+
/// emits no sum terminal and no blend. A sentence is not a fence, so this
173+
/// sweeps every opcode a caller can put in an `LgjOpDesc` — the nine
174+
/// defined ones and a band of undefined ones either side — through the
175+
/// real `lower_plan` and reads every emitted predicate.
176+
///
177+
/// **When LGJ gains a Range lowering this test goes RED**, and that is its
178+
/// whole purpose: it forces the author to decide, deliberately and
179+
/// visibly, what a caller should see when a range leaves the lane — rather
180+
/// than inheriting `LGJ_ERR_ALLOCATION_FAILED`, which would then be a lie.
181+
#[test]
182+
fn no_opcode_lowers_to_pred_range() {
183+
let mut lowered_count = 0usize;
184+
for op in 0u32..=32 {
185+
for combine in [LGJ_COMBINE_AND, LGJ_COMBINE_OR] {
186+
let desc = LgjOpDesc {
187+
op,
188+
lane_id: 0,
189+
operand: -1,
190+
combine,
191+
_reserved: 0,
192+
};
193+
let Some(lowered) = lower_plan(&[desc]) else {
194+
continue;
195+
};
196+
let Lowered::Program(program) = lowered else {
197+
continue; // AllRows: no predicate at all.
198+
};
199+
lowered_count += 1;
200+
for mask_op in &program.ops {
201+
if let MaskOp::Pred { pred, .. } = mask_op {
202+
assert!(
203+
!matches!(pred, Pred::Range { .. }),
204+
"opcode {op} lowered to Pred::Range — `exec_error_to_status` maps \
205+
`RangeOutOfBounds` as an internal bug, which is now false: give it \
206+
a caller-visible status before shipping the Range lowering"
207+
);
208+
}
209+
}
210+
}
211+
}
212+
// Anti-vacuity: a sweep that lowers nothing proves nothing. Nine
213+
// opcodes are defined and each lowers under `LGJ_COMBINE_AND`.
214+
assert!(
215+
lowered_count >= 9,
216+
"the sweep lowered only {lowered_count} programs — it is not exercising the lowering"
217+
);
218+
}
219+
}

0 commit comments

Comments
 (0)