diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 31ac79589..c934024c3 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,9 @@ +## 2026-10-14 — D-MTP-0 harvest: six substrate-mapper runs, one reply each + +- Card `substrate-mapper` (run as a general-purpose agent with the card pasted; the card is not registered in this session), six units: `hyperon-atom`, `hyperon-space`, the minimal-MeTTa interpreter, the grounded stdlib, lance-graph `nal`/`r2il`/`relalg`, `quack`. +- Ledger: spot-checks `Space::query` L175, `complex_query` L340, `GroundingSpace` L17, interpreter dispatch L417/L453, `query` L604, `FoldDialect` L260 — all correct; 0 wrong. 1 missed: no map recorded that `ModuleSpace` splits conjunctions and requires `ModuleSpace` dependencies (`space/module.rs:33-48`); the orchestrator found it by reading the file in full. +- Cost: about 515k–646k subagent tokens per run. No conclusion came from a card. + ## 2026-10-14 — D-HOLO H6b harvest: four substrate-mapper runs, one reply each - Card `substrate-mapper`, four units: `holo_interval_probe.rs` (H6), mask-risc exec, mask-risc IR, ndarray `simd` mask re-exports. diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index a718e5bb1..666580bb7 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,11 @@ +## 2026-10-14 (2) — metta-polyfill-v1 — Hyperon/MeTTa as a front end over the masking-algebra polyfill → `.claude/plans/metta-polyfill-v1.md` + +**Status:** PROPOSAL; D-MTP-0 (map + measurements), D-MTP-1 (join probe, measured), D-MTP-1b (range hop, measured) and D-MTP-6 (datafrog oracle) done; D-MTP-2..5 not authorized. + +- MeTTa is a declaration IR, not an execution IR: the Datalog subset lowers through `lance-graph-relalg` → R2IL → mask-risc; everything else stays in Hyperon's interpreter. +- Seam: a named `LanceSpace` implementing `hyperon_space::Space`; `ModuleSpace` splits conjunctions, so `&self` is not the seam. +- Measured PeTTa (SWI-Prolog) near-linear, Hyperon superlinear (anc_800: 0.42 s vs 417 s); new trie panic at ≥ 1,024 children. + ## 2026-10-14 — d-holo-address-indexed-algebra-v1 — falsifiers H1, H5, H6 for the address-indexed algebra → `.claude/plans/d-holo-address-indexed-algebra-v1.md` **Status:** PRE-REGISTERED; nothing measured. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index af046bf12..bd0c62540 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,18 @@ +## D-MTP — Hyperon/MeTTa over the masking-algebra polyfill (2026-10-14) + +Plan: `.claude/plans/metta-polyfill-v1.md`. Proposed crate `lance-graph-metta` (not created). + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| **D-MTP-0** | Adoption map, MeTTa-as-IR verdict, PeTTa vs Hyperon measurements, trie panic ≥ 1,024 children | Done 2026-10-14 (entry `2026-10-14-metta-polyfill-map.md`) | — | +| **D-MTP-1** | Two-hop join lowered by hand through Quack `Semijoin` (relalg has no hop), n = 10²..10⁶, v4 | Measured 2026-10-14: pairs == datafrog at every n, counts == Hyperon/PeTTa; at 10⁶ emit 4.0–5.7 ns/pair vs datafrog 59–71 (7.2–9.6× with ingest charged per emitted pair); count 1.2–1.8 ns/row, the random `Gather` read, not sub-ns as expected | == oracles; two disable runs red as pre-registered | +| **D-MTP-1b** | The hop as a range: preimage of `valid`'s runs under a monotone `dst` vs the gather, clustered and scattered | Measured (§6d): R-declared count 0.0001–0.0004 ns/row vs gather 1.3–2.4 at 10⁶ | == datafrog; two disable runs red; costs reported | +| **D-MTP-2** | `LanceSpace` (`Space + SpaceMut`) as a named space for the non-recursive Datalog subset; golden corpus rerun | Queued (not authorized) | 32/33 golden files still pass; refusals named; whole-conjunction vs per-conjunct calls recorded | +| **D-MTP-3** | Recursive Datalog rules → relalg μ through the external driver d′ | Queued (not authorized) | `anc_n` answers identical; no fixpoint state in mask-risc | +| **D-MTP-4** | Fork fix for the expression-width trie panic (≥ 1,024 children), test first; distinct from the id-decoding fix `1ffdbd6` | Queued (needs operator's word) | red before, green after | +| **D-MTP-6** | datafrog as the exact Rust oracle (probe/test only) | Measured 2026-10-14 (v4): join 10⁶ in 65–69 ns/pair, seeded ancestors 10⁶ in 286–298 ns/tuple; counts match; not yet a committed probe | bag-of-pairs equality with Hyperon and PeTTa | +| **D-MTP-5** | PeTTa as second oracle in the probes | Queued (not authorized) | PeTTa/Hyperon divergences recorded, never averaged | + ## D-RA — Relational algebra over coordinate populations, a D-CMR prototype (2026-10-13) Plan: `.claude/plans/relalg-v1.md`. Crate: `crates/lance-graph-relalg` (own workspace root, excluded). Prototype for D-CMR-6 rules over residual comparisons and a lowering through R2IL. diff --git a/.claude/board/entries/2026-10-14-d-mtp-1-join.md b/.claude/board/entries/2026-10-14-d-mtp-1-join.md new file mode 100644 index 000000000..faaa83d1c --- /dev/null +++ b/.claude/board/entries/2026-10-14-d-mtp-1-join.md @@ -0,0 +1,20 @@ +# 2026-10-14 — D-MTP-1 — the MeTTa two-hop join through Quack Semijoin (measured) + +**Status:** MEASURED (v4, `avx512f=true`, release, best of 5, three runs). Plan +`.claude/plans/metta-polyfill-v1.md` §6a (pre-registration) and §6b (results). Probe +`crates/lance-graph-metta/examples/join_probe.rs`. + +**Outcome:** +- `(match &kb (, (parent $x $y) (parent $y $z)) ($x $z))` lowers to + `Plane(valid) AND Semijoin { fk: dst, foreign: valid }` → one mask-risc program + (`Gather`); pairs are emitted only over the kept bits. `lance-graph-relalg` has no hop, + so the plan's "through relalg" was corrected to Quack. +- Pairs == datafrog (fork) at n = 10²..10⁶ on a total (F1) and a partial (F2) relation; + counts == Hyperon and PeTTa. Two disable runs red exactly as pre-registered. +- At 10⁶: emission 4.0–5.7 ns/pair against datafrog 59–71 ns/pair; with ingest charged + per emitted pair, 6.5–7.1 ns (F1) and 8.6–9.2 ns (F2, 666,666 facts for 476,190 pairs). The folded count is 1.2–1.8 ns/row, not the sub-nanosecond the plan + expected: the cost is the dependent random read of the gather, not the mask algebra. + +**Open:** a layout where the hop is a coordinate range (no per-row gather); one-to-many +relations; recursion (D-MTP-3); the `Space` adapter (D-MTP-2); symbol → address +resolution cost. diff --git a/.claude/board/entries/2026-10-14-d-mtp-1b-range-hop.md b/.claude/board/entries/2026-10-14-d-mtp-1b-range-hop.md new file mode 100644 index 000000000..dfc7f573e --- /dev/null +++ b/.claude/board/entries/2026-10-14-d-mtp-1b-range-hop.md @@ -0,0 +1,19 @@ +# 2026-10-14 — D-MTP-1b — the hop as a range (measured) + +**Status:** MEASURED (v4, `avx512f=true`, release, best of 5, three runs). Plan +`.claude/plans/metta-polyfill-v1.md` §6c (pre-registration) and §6d (results). Probe +`crates/lance-graph-metta/examples/join_range_probe.rs`. + +**Outcome:** +- With `dst` non-decreasing over rows (checked once at ingest), the two-hop join's kept rows + are `valid`'s runs intersected with their preimages under `dst` (`partition_point`). + Pairs == datafrog on a BFS tree (F3), the same tree with every third level removed (F4) + and both under a random address permutation; the range path refuses the permuted lane. +- At 10⁶ the count costs 0.0001–0.0004 ns/row with runs recorded at ingest (O(runs·log n), + 0.1–0.4 µs total) and 0.04–0.05 ns/row when the runs are scanned from the plane, against + 1.3–2.4 ns/row for the gather program on the same data. Pair emission is 1.2–2.0 ns/pair. +- Clustering alone (same gather, local reads) saves only 11–36 %; the win comes from the + hop being a range, not from locality. + +**Open:** the order witness is probe-local (the contract's `OrderedLaneWitness` covers sealed +facet lanes only); a chained range for the second hop; scattered layouts keep the gather. diff --git a/.claude/board/entries/2026-10-14-metta-polyfill-map.md b/.claude/board/entries/2026-10-14-metta-polyfill-map.md new file mode 100644 index 000000000..961110664 --- /dev/null +++ b/.claude/board/entries/2026-10-14-metta-polyfill-map.md @@ -0,0 +1,29 @@ +# 2026-10-14 — D-MTP-0 — where Hyperon/MeTTa attaches to the masking-algebra polyfill + +**Status:** MEASURED (engine comparison, v4 host, single whole-run timings) + VERIFIED-IN-CODE +(Hyperon fork `1ffdbd6`). Plan `.claude/plans/metta-polyfill-v1.md`. No crate, no code. + +**Outcome:** +- MeTTa works as a declaration IR (rules, facts, queries as typed S-expressions, names resolved + once to addresses), not as the execution IR. Execution: MeTTa Datalog subset → `lance-graph-relalg` + → R2IL `FoldDialect` → mask-risc. Term construction, bags, grounded ops, types and state stay in + Hyperon's unchanged interpreter. +- Seam: `hyperon_space::Space::query` (`hyperon-space/src/lib.rs:175`). A space queried directly by + `match` gets the whole `(, …)` pattern (`stdlib/core.rs:163`); `ModuleSpace` splits it into single + conjuncts first and panics on a non-`ModuleSpace` dependency (`space/module.rs:33-48`). So the + adapter is a named space, never `&self` or a module dependency. +- Measured, same generated programs: two-hop join PeTTa 0.29 / 0.41 / 1.47 s vs Hyperon 0.11 / 5.9 s / + panic at n = 10² / 10³ / 10⁴; recursive ancestor PeTTa 0.29 / 0.33 / 0.42 s vs Hyperon + 0.28 / 7.4 / 417 s at n = 50 / 200 / 800. Answers correct wherever an engine finished. +- New panic, separate from the id-decoding fix already on the fork (`1ffdbd6`, branch + `fix-trie-key-value`, on which these runs were taken): an expression with ≥ 1,024 children passed to a MeTTa function aborts in + `TrieKey::start_expr` (`hyperon-space/src/index/trie.rs:539`, `TK_MAX_EXPRESSION_SIZE = 1 << 10`); + 1,023 children works. This stopped the n = 10⁴ join. +- Rust Datalog oracle: datafrog 2.0.1 from the fork `AdaWorldAPI/datafrog` (v4, best of 5) runs join 10⁶ in 65–69 ms (65–69 ns/pair) and the + seeded ancestor query 10⁶ in 0.29 s; join_10000 in 0.5 ms against PeTTa 1.47 s and Hyperon > 600 s. + Oracle and comparison bar only, never a production engine (hash/sort joins beside mask-risc). +- Crate name: `lance-graph-metta` (one concern: front end + `Space` adapter); "cog" would mix in + PLN, the miner, attention and Classic AtomSpace (AGPL). + +**Open:** whether `match &self` can reach whole conjunctions without patching Hyperon; D-MTP-1..5 +not authorized; the expression-width panic has no fix yet. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index c8bf108ca..b575ceba5 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,13 +25,16 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -331 entries, 2026-08-06 .. 2026-10-14. +334 entries, 2026-08-06 .. 2026-10-14. | date | entry id | finding | file | |---|---|---|---| | 2026-10-14 | `revision-layer-invariants` | | [2026-10-14-revision-layer-invariants.md](2026-10-14-revision-layer-invariants.md) | +| 2026-10-14 | `D-MTP-0` | where Hyperon/MeTTa attaches to the masking-algebra polyfill | [2026-10-14-metta-polyfill-map.md](2026-10-14-metta-polyfill-map.md) | | 2026-10-14 | `ledger-revision-after-1553` | | [2026-10-14-ledger-revision-after-1553.md](2026-10-14-ledger-revision-after-1553.md) | | 2026-10-14 | `D-NALP-4` | | [2026-10-14-isa-truth-exact-codes.md](2026-10-14-isa-truth-exact-codes.md) | +| 2026-10-14 | `D-MTP-1b` | the hop as a range (measured) | [2026-10-14-d-mtp-1b-range-hop.md](2026-10-14-d-mtp-1b-range-hop.md) | +| 2026-10-14 | `D-MTP-1` | the MeTTa two-hop join through Quack Semijoin (measured) | [2026-10-14-d-mtp-1-join.md](2026-10-14-d-mtp-1-join.md) | | 2026-10-14 | `D-HOLO-H6b` | normal forms on the Context × Operation tile (measured) | [2026-10-14-d-holo-h6b.md](2026-10-14-d-holo-h6b.md) | | 2026-10-14 | `D-HOLO` | | [2026-10-14-d-holo-falsifiers.md](2026-10-14-d-holo-falsifiers.md) | | 2026-10-13 | `D-PROD-V4` | | [2026-10-13-production-targets-v4.md](2026-10-13-production-targets-v4.md) | diff --git a/.claude/plans/metta-polyfill-v1.md b/.claude/plans/metta-polyfill-v1.md new file mode 100644 index 000000000..351de5def --- /dev/null +++ b/.claude/plans/metta-polyfill-v1.md @@ -0,0 +1,402 @@ +# metta-polyfill-v1 — Hyperon/MeTTa over the masking-algebra polyfill + +**Status:** PROPOSAL (D-MTP-0..5). Nothing beyond D-MTP-0 (this map and its +measurements) is authorized. No crate exists yet. +**Date:** 2026-10-14. +**Repos:** lance-graph (`lance-graph-relalg`, `lance-graph-r2il`, +`lance-graph-mask-risc`, `lance-graph-quack`, `lance-graph-nal`), +`AdaWorldAPI/hyperon-experimental` (fork, MIT), `AdaWorldAPI/petta` (MIT). +**Parents:** `.arxiv/2026-10_hyperon_metta_to_lance_graph_parity_map.md` +(guardrails, HY-Q01..15, probes P0–P7), `nal-polyfill-v1.md` (the template: +text front end → bind → procedures → `FoldDialect` → mask-risc), +`lance-graph-relalg` (#1546, the describing IR). + +## 1. The question + +Can MeTTa serve as an IR for the masking algebra, and where does +hyperon-experimental attach so that the atoms MeTTa uses are served by the +polyfill rather than by Hyperon's own trie? Should the crate be +`lance-graph-cog` or `lance-graph-metta`? + +## 2. What Hyperon is, at the seams (VERIFIED-IN-CODE, fork `1ffdbd6`) + +- **Atoms:** `Symbol`, `Variable`, `Grounded`, `Expression` + (`hyperon-atom/src/lib.rs`), matched by `match_atoms` / `Bindings` + (`matcher.rs`). +- **Interpreter:** minimal MeTTa is one dispatch over 12 instruction symbols + (`lib/src/metta/interpreter.rs:415-458`: `eval`, `evalc`, `chain`, + `function`, `collapse-bind`, `unify`, `decons-atom`, `cons-atom`, + `superpose-bind`, `metta`, `context-space`, `call-native`; `function` is an + internal state that panics if reached). The full language (types, `let`, + `if`, `case`, …) is MeTTa code in `stdlib.metta` plus grounded ops on top. +- **The space seam:** `hyperon_space::Space::query(&self, &Atom) -> + BindingsSet` (`hyperon-space/src/lib.rs:175`) and `SpaceMut` + (`add`/`remove`/`replace`, `:219`). The interpreter reaches data only + through `query` (`interpreter.rs:604`, `eval_impl` → `query`). +- **Conjunctions:** `GroundingSpace::query` (`lib/src/space/grounding/mod.rs:147`) + calls `complex_query` (`hyperon-space/src/lib.rs:340`), which splits + `(, q1 q2 …)` and joins **tuple at a time**: for every binding of the + conjuncts so far it substitutes into the next conjunct and probes the trie + again (`:345-368`). The trie (`index/trie.rs`) is the only index. + +## 3. Measured (2026-10-14, v4 host, whole-run seconds; engine comparison, not a primitive timing) + +Two Datalog-shaped MeTTa programs, generated, identical for both engines: +`join_n` = n facts `(parent nᵢ n₍₇ᵢ₊₃₎ mod n)` and the two-hop conjunctive +query `(match &self (, (parent $x $y) (parent $y $z)) ($x $z))`, counted; +`anc_n` = a chain of n−1 `parent` facts and the recursive ancestor rule, +`(anc n0 $z)` counted. Both engines return the correct counts wherever they +finish (100 / 1,000 / 10,000 pairs; 49 / 199 / 799 ancestors). + +| program | PeTTa (SWI-Prolog 10.0.2), total | Hyperon (fork, release, native), run | +|---|---|---| +| startup only (`!(+ 1 1)`) | 0.32 | 0.000 (init 0.016–0.020) | +| join_100 | 0.29 | 0.11 | +| join_1000 | 0.41 | 5.92 | +| join_10000 | 1.47 | **panic** (`trie.rs:539`, see §3.1) | +| anc_50 | 0.29 | 0.28 | +| anc_200 | 0.33 | 7.37 | +| anc_800 | 0.42 | 416.9 | + +PeTTa compiles MeTTa to Prolog clauses and stays near its startup cost; Hyperon +grows about n^1.7 on the join (×54 for ×10) and n^2.4–2.9 on the recursion +(×26 then ×56 per ×4). Single runs; the ratios are far outside the 40 % noise +band, the absolute seconds are not a primitive claim. + +### 3.0 A Rust Datalog oracle (datafrog 2, measured 2026-10-14) + +`datafrog` (the semi-naive engine behind Polonius), consumed from the fork +`AdaWorldAPI/datafrog` (`master`, `edd7cfc`, version 2.0.1), on the same generated programs, release, +`target-cpu=native`, program-printed tier `avx512f=true`, best of 5, three runs +within 5–15 %. Relation construction (sorting) is charged. Counts asserted +against the known answers, which match Hyperon's and PeTTa's. + +| program | datafrog | per output tuple | +|---|---|---| +| join, n = 10² / 10⁴ / 10⁶ | 3 µs / 0.51–0.62 ms / 65–69 ms | 33 / 51–62 / 65–69 ns | +| ancestors from n0 (the MeTTa query, seeded), n = 800 / 10⁴ / 10⁶ | 0.14 ms / 2.2 ms / 0.29 s | 179–185 / 222 / 286–298 ns | +| full closure on the chain, n = 2,000 (2.0 M tuples) | 0.28 s | 139–143 ns | + +It finishes join_10000 in 0.5 ms where Hyperon does not finish in 600 s and +PeTTa needs 1.47 s (mostly its 0.32 s startup plus compilation). The seeded +ancestor query costs more per tuple than the join because a chain derives one +tuple per semi-naive round, so per-round overhead dominates. These are +per-tuple engine costs on hash-free sorted merges, not mask-risc primitives; +they are the bar a folded lowering is compared against, not a production path. +First run against the crates.io 2.0.1 release, re-run against the fork: every count +identical, timings within the noise band (join 10⁶ 65.0 ns/pair; seeded ancestors +10⁶ 0.31 s). Probe source: local crate `/home/user/datalog-oracle-probe` (not committed). + +### 3.1 A second Hyperon trie bug (distinct from the fix already on the fork) + +The measurements above ran on the fork's `fix-trie-key-value` branch (`1ffdbd6`), +which already carries the earlier fix: `TrieKey::value()` decoded ids ≥ 1,024 +wrongly (`& (MASK - 1024)` instead of `- 1024`), so a space with more than 1,024 +distinct atoms returned wrong atoms or panicked. That fix is in place and is not +what follows. The panic below is a separate limit that shares the constant 1,024. + +Any expression with 1,024 or more children passed to a MeTTa-defined function +panics: `(let $r (0 … 1022) (size-atom $r))` returns 1023; the same with 1,024 +items aborts with `assertion failed: size < TK_MAX_EXPRESSION_SIZE` +(`hyperon-space/src/index/trie.rs:539`, `TrieKey::start_expr`; +`TK_MAX_EXPRESSION_SIZE = 1 << 10`, `:511`). The interpreter queries the +space for `(= (let $r …) $X)`, and the trie cannot key an expression +that wide. `(size-atom (0 … 1023))` called directly does not panic, because a +grounded op is executed without a space query. This is what stopped join_10000 +(its `collapse` result has 10,000 children). Not fixed here; reproduced on `1ffdbd6`. + +## 4. Can MeTTa work as an IR? + +**As a declaration IR: yes. As the execution IR: no.** + +- MeTTa expressions are typed S-expressions over four atom kinds; rules + (`=`), facts and queries (`match`, `(, …)`) are data. That is what the + stack needs from a front end: text parsed once, names resolved through the + label codebook to addresses (nal-polyfill §4 H), nothing after the binder + holding a string. +- Minimal MeTTa is an interpreter for nondeterministic, tuple-at-a-time + rewriting (`superpose-bind`, `collapse-bind`, `cons-atom`, + `decons-atom`, `call-native`). It constructs new terms and runs grounded + host code. That is the opposite of a folded mask program, and §3 shows what + executing it naively costs. +- So the arrangement is the one NAL and SQL already use: **MeTTa is a + front end; `lance-graph-relalg` describes; R2IL / `FoldDialect` executes; + mask-risc is the only engine.** Three tiers of the language map to three + outcomes: + +| MeTTa fragment | lowers to | outcome | +|---|---|---| +| facts in a space, `match` over one pattern, `(, …)` conjunctions with shared variables, no new terms | `Rel` selections per conjunct, joins on shared variables (`EqU32Via` / `Gather`), one fold, bindings only at the terminal | **polyfill** (one program per query, not one trie probe per binding) | +| recursive `=` rules that are Datalog (no function symbols in heads, no grounded calls) | relalg `μ` driven by the external driver d′ (`2026-10-03-coresearch-chained-hop.md`: recursion stays outside mask-risc), each iteration one fold | **polyfill, gated** (D-MTP-3) | +| term construction, `superpose`/`collapse` bags, grounded ops, dependent types, state | Hyperon's own interpreter, unchanged | **stays in Hyperon** (typed refusal on our side, never approximated) | + +## 5. Where it attaches + +```text +MeTTa text ─► hyperon parser ─► Hyperon interpreter (fork, unchanged, 12 ops) + │ Space::query(&Atom) / SpaceMut + ▼ + lance-graph-metta: LanceSpace (impl Space + SpaceMut) + atoms → addresses (label codebook), facts → rows + query atom → Rel (σ per conjunct, ⋈ on shared vars) + ▼ + lance-graph-relalg (normalize, symbolic ∅ / count) + ▼ + lance-graph-r2il FoldDialect ─► mask-risc ─► ndarray::simd +``` + +- **The seam is `Space`, not the interpreter.** `complex_query` is a helper a + space chooses to call, not part of the trait, so a `LanceSpace` that is + queried directly receives the whole `(, q1 q2 …)` atom and can lower the full + conjunction into one program. +- **But only as a named space, never inside `&self`.** `ModuleSpace` + (`lib/src/space/module.rs:33-48`) runs `complex_query` itself and hands its + main space and every dependency one conjunct at a time; a dependency that is + not a `ModuleSpace` panics (`:44`). So `LanceSpace` is bound as its own token + (`!(bind! &kb (new-lance-space …))`, `(match &kb (, …) …)`), and the module's + `&self` stays an ordinary `GroundingSpace` holding rules, grounded atoms and + everything outside the Datalog subset. `match` hands its pattern unchanged to + the named space's `query` (`lib/src/metta/runner/stdlib/core.rs:163`). + Wrapped in a `ModuleSpace`, it would degrade to one fold per conjunct per + binding, which is the cost §3 measures. Whether `match &self` can reach the + conjunction without patching Hyperon is open (D-MTP-2). +- **Recursive rules live in the named space too.** The interpreter evaluates + `=` rules by querying the context space (`interpreter.rs:604`), i.e. `&self`, + one call per reduction step. A recursive Datalog rule left there never + reaches the polyfill. D-MTP-3 therefore stores the rules as algebra values in + `LanceSpace` and answers a pattern over the derived relation + (`(match &kb (anc n0 $z) $z)`) with the closure computed by d′, not by + `=` rewriting. +- **Atoms are addresses, not strings.** A `Symbol` resolves once to a + `classid × NodeGuid` (or a term id within a class); an `Expression` fact of + fixed shape `(functor a₁ … aₖ)` is a row of a class keyed by the functor and + arity, one lane per argument. `Variable` never reaches a row. Shapes the + class does not declare are refused, never stored as text. +- **Multiplicity is part of the answer.** `BindingsSet` is a bag; the + terminal must emit one `Bindings` per match (the `b4_nondeterm` kill + condition in the parity map). A row mask alone is not an answer. +- **The NAL polyfill is the precedent and stays separate.** PeTTa's + `lib_nars.metta` / `lib_pln.metta` are oracles; NAL operations go through + `lance-graph-nal` (`nal::{Op}`, global core methods, no classid plug), not + through MeTTa rules. + +## 6. `lance-graph-cog` or `lance-graph-metta` + +Recommendation: **`lance-graph-metta`**, workspace-excluded like +`lance-graph-relalg`, depending on the fork's `hyperon-atom` and +`hyperon-space` only (not the interpreter crate). + +- It holds exactly one concern: the MeTTa front end and the `Space` adapter. +- "cog" would pull in OpenCog Classic AtomSpace, PLN, the miner and + attention, which the parity map assigns elsewhere (#1497 owns + scheduling/attention; Classic AtomSpace is AGPL and a different engine). +- An earlier note proposed a separate `metta-rs` repo. That fits a full + transcode of the interpreter; this plan does not transcode it, so a crate + beside relalg is enough. Revisit if the interpreter itself is ever ported. + +## 6a. D-MTP-1 pre-registration (2026-10-14, before any code) + +**Correction to §4:** `lance-graph-relalg` cannot lower the join: its own +crate docs say "recursion (`μ`) and the hop (preimage) are not built". The +existing join surface is Quack's `Filter::Semijoin { fk, foreign }`, which +lowers to mask-risc `MaskOp::Gather` over a RESIDENT foreign plane. D-MTP-1 +uses that path; relalg gains the hop later or not at all. + +**Layout (ingest, charged separately).** Symbols resolve once to addresses. +The `parent` facts become a table over the address space of their first +argument: row `a` holds the fact whose subject has address `a`, the `dst` lane +holds the object's address (a coordinate, not a key), and a `valid` plane marks +the addresses that have a fact. Functional relations only: one fact per +subject. A second fact for the same subject is refused at ingest, not +approximated (one-to-many needs `ScatterOrU32`/a CSR and is out of scope here). + +**Query.** `(match &kb (, (parent $x $y) (parent $y $z)) ($x $z))` lowers to +`Filter::and([Plane(valid), Semijoin { fk: dst, foreign: valid }])`. Two +terminals: `Agg::Count` (the MeTTa `size-atom` of the collapsed result) and +`Agg::Rows` (`Keep`) followed by binding emission `(x, dst[dst[x]])` over the +kept bits only — the one place pairs are materialized. + +**Fixtures** (generated, identical in all engines): +- F1 `join_n`: `parent(i, (7i+3) mod n)` for every `i` (total, every row + survives; count = n). +- F2 `join_n_partial`: the same, only for `i mod 3 ≠ 0` (the semijoin rejects + rows whose target has no fact; count < n). +- n = 10², 10³, 10⁴, 10⁵, 10⁶. + +**Oracles.** datafrog (fork) at every n: the emitted pairs as a sorted vector +must equal datafrog's, not only the count. Hyperon and PeTTa on F1 and F2 at +n = 10², 10³ (and PeTTa 10⁴): counts must match. + +**Disable runs** (commit first, assert each patch applied once): +1. drop the `Semijoin` leaf → F2 count and pairs must go red (F1 stays green, + which is why F2 exists); +2. emit `(x, dst[x])` instead of `(x, dst[dst[x]])` → pairs red on F1 and F2. + +**Measured, reported, no threshold claimed before the run** (v4 arm, +program-printed tier, best of 5): ingest ns/fact; count program ns/row; +Rows + emission ns/pair; datafrog ns/pair beside it (its relation build +charged, as in §3.0). Expected only as a direction, not a gate: the folded +count costs a small fraction of a nanosecond per row, emission a few ns per +pair, both below datafrog's 65 ns/pair, because ingest has already turned +the join key into a coordinate. If ingest is charged to the query, the +comparison is reported both ways. + +**Home.** First probe of `crates/lance-graph-metta` (own workspace root, +excluded, like `lance-graph-relalg`), so the datafrog git dev-dependency never +enters the main workspace's resolution. + +## 6b. D-MTP-1 results (2026-10-14) + +Probe `crates/lance-graph-metta/examples/join_probe.rs` (5 tests, run as part +of `cargo test`). v4 host, program-printed `avx512f=true`, release, best of 5, +three runs; ranges below span the three. + +**Correctness.** The emitted pairs equal datafrog's as sorted vectors at every +n on F1 and F2. Counts equal Hyperon (F1 100 / 1,000; F2 46 / 476) and PeTTa +(F2 46 / 476 / 4,762). Disable runs, after committing: dropping the +`Semijoin` leaf fails F2 and the pinned counts while F1 stays green; emitting +`(x, dst[x])` fails the pairs on F1 and F2. Both as pre-registered. + +| fixture, n = 10⁶ | ingest ns/fact | count ns/row | Rows + emit ns/pair | datafrog ns/pair | +|---|---|---|---|---| +| F1 (10⁶ pairs) | 2.5–3.0 | 1.72–1.79 | 3.97–4.06 | 59–62 | +| F2 (476,190 pairs) | 2.3–2.5 | 1.21–1.24 | 5.4–5.7 | 66–71 | + +From 10³ to 10⁶ the count stays at 1.1–2.4 ns/row and emission at 3.0–5.7 +ns/pair; at n = 100 fixed costs (lowering, scratch) dominate (8–9 ns/row, +10–21 ns/pair). + +**Against the pre-registration.** +- Pairs and counts identical: met. +- Emission below datafrog: met, by 11–15× per pair at 10⁶. With ingest + charged to the query, normalized per emitted pair (F2 ingests 666,666 facts + for 476,190 pairs, so its ingest counts 1.4×): F1 6.5–7.1 ns/pair against + 59–62 (8.4–9.6×), F2 8.6–9.2 ns/pair against 66–71 (7.2–8.2×). +- "The folded count costs a small fraction of a nanosecond per row": NOT met. + It costs 1.2–1.8 ns/row. The semijoin is a `Gather` through `dst`, one + dependent random read per row (`(7i+3) mod n` scatters the targets), and + that read, not the mask algebra, sets the cost. A layout where the hop is a + range (rule: hops are coordinate ranges, chained hops are intersections) is + what would remove it; this fixture has no such locality to exploit. + +**Scope.** Functional relations over a dense address space, integer facts +given to every engine. Symbol → address resolution through the label codebook +is not charged to anyone here; one-to-many relations, recursion and the +`Space` adapter are not covered. + +## 6c. D-MTP-1b pre-registration: the hop as a range (2026-10-14, before any code) + +**Question.** D-MTP-1's count costs 1.2–1.8 ns/row because the semijoin is a +`Gather`: one dependent random read per row. When the link lane is monotone in +row order, can the hop become interval arithmetic instead (the hop as a +coordinate range, chained hops as intersections)? + +**Mechanism.** If `dst` is non-decreasing over rows, the rows whose target lies +in `[a, b)` are exactly `[pp(dst < a), pp(dst < b))` (`pp` = `partition_point`, +as in `teleport_probe`). So `Semijoin(dst, valid)` restricted to rows in +`valid` is the intersection of two interval sets: `valid`'s runs, and the +preimages of `valid`'s runs. The count is the length of that intersection; +the pairs `(x, dst[dst[x]])` are emitted by walking the intervals. + +**Precondition, checked not assumed.** Monotonicity of `dst` is verified once at +ingest (an O(n) pass, charged to ingest) and recorded as a flag; the range path +refuses a lane without it and the caller falls back to the gather. This is a +probe-local witness, not the contract's `OrderedLaneWitness` (which covers +sealed facet lanes only, `teleport-probe` OPEN item). + +**Fixtures** (generated, n = 10³..10⁶): +- F3 BFS tree, fanout 4: `parent(x, (x-1)/4)` for every `x ≥ 1` (root has no + fact); `dst` monotone; `valid` is one run. +- F4: F3 without the facts of nodes whose depth is a multiple of 3; `valid` is + one run per kept level band (about depth·2/3 runs), so both the source and the + target restriction bite. +- F3s / F4s: the same facts under a random permutation of addresses (the + scattered control; no range path exists there). + +**Arms.** +- G-scattered: D-MTP-1's gather program on F3s/F4s. +- G-clustered: the same gather program on F3/F4 (same algebra, local reads). +- R-scan: `valid`'s runs extracted from the plane by a word scan, then the + interval path. +- R-declared: `valid`'s runs recorded at ingest (no scan), then the interval path. + +**Oracle and tests.** datafrog pairs (sorted) equal every arm's pairs at every +n; counts equal. The range path refuses a non-monotone `dst`. Disable runs +(commit first, assert each patch applied once): (1) intersect with the +preimage runs only, skipping `valid`'s own runs → F4 red, F3 green; (2) take +`valid`'s runs as the answer, skipping the preimage → F4 red. + +**Expected, as a direction only (no threshold before the run).** R count costs +O(runs · log n), far below a nanosecond per row at 10⁶; G-clustered sits +between G-scattered and R; pair emission on the range path is sequential and +should not exceed G-clustered's. Reported per row and per pair, v4, +program-printed tier, best of 5, three runs; ingest (including the +monotonicity pass and, for R-declared, the run recording) reported separately. + +## 6d. D-MTP-1b results (2026-10-14) + +Probe `crates/lance-graph-metta/examples/join_range_probe.rs` (3 tests, shared +fixture code in `examples/common/mod.rs`). v4 host, program-printed +`avx512f=true`, release, best of 5, three runs; ranges span the three. + +**Correctness.** Every arm's pairs equal datafrog's (sorted) on F3, F4 and the +scattered twins at n = 100, 1,000, 4,097; counts equal at every n up to 10⁶. +The range path refuses the scattered (non-monotone) lane. Disable runs, after +committing: (1) taking the preimage of `valid`'s runs without intersecting +with `valid` itself fails F4 (`banded=true`) while F3 stays green; (2) taking +`valid`'s runs without the preimage fails already on F3. Both as +pre-registered. A fixture test pins that both halves of the intersection +change the answer on F4. + +| n = 10⁶ | G-scattered count ns/row | G-clustered count ns/row | R-scan count ns/row | R-declared count ns/row | emit ns/pair G-clustered / R | datafrog ns/pair | +|---|---|---|---|---|---|---| +| F3 (999,995 pairs, 1 run) | 1.96–3.13 | 1.74–1.99 | 0.037–0.055 | 0.00010–0.00011 | 3.4–5.2 / 1.5–1.8 | 16–24 | +| F4 (66,576 pairs, 4 runs) | 1.52–2.43 | 1.27–2.08 | 0.037–0.051 | 0.00026–0.00038 | 21–35 / 1.2–2.0 | 144–196 | + +Ingest is charged separately: 3.4–7.1 ns/row clustered, of which the +monotonicity pass is 0.5–0.9 ns/row (scalar `windows(2)`), and 7.8–10.2 ns/row +scattered. G-clustered's emission per pair on F4 is high because the gather +pass runs over all 10⁶ rows for 66,576 pairs. + +**Against the pre-registration.** +- Pairs identical; two disable runs red: met. +- "R count far below a nanosecond per row": met. R-declared is + O(runs · log n), about 0.1–0.4 µs for the whole count at 10⁶, 3,000–24,000× + below the gather. R-scan pays the word scan of the plane (n/64 words) and + sits at 0.04–0.05 ns/row. +- "G-clustered between G-scattered and R": met only barely. Clustering the + same gather program saves 11–36 %; the gather's per-row work, not its cache + misses, sets most of its cost at this size. +- "R emission not above G-clustered": met (1.2–2.0 ns/pair; the remaining + cost is the dependent read `dst[dst[x]]`). +- The fixture note "about depth·2/3 runs" was wrong: adjacent kept levels + merge, so F4 has one run per band of two levels (2 runs at 4,097, 4 at 10⁶). + +**Scope.** The range path needs the order witness; it is a probe-local flag +set at ingest, not the contract's `OrderedLaneWitness`, which covers sealed +facet lanes only. A scattered layout (F3s/F4s) has no range path and keeps +the gather. Pair emission still reads `dst` twice per pair; a chained range +(the second hop as a range too) is not built here. + +## 7. Deliverables + +| D-id | scope | gate | +|---|---|---| +| D-MTP-0 | this map, measurements §3, bug §3.1 | — (done) | +| D-MTP-1 | pre-registered probe (§6a): the two-hop join lowered by hand through Quack `Semijoin` (relalg has no hop) against datafrog, Hyperon and PeTTa, n = 10²..10⁶, v4 | answers identical (count and bag of pairs); cost reported, no threshold claimed before the run — **measured, §6b** | +| D-MTP-2 | `LanceSpace` adapter (`Space + SpaceMut`) for the non-recursive Datalog subset, bound as a named space; Hyperon's golden corpus (`hyperon-probe/golden`, 32 of 33 passing) rerun with facts moved into it | every golden file that passes today still passes; refusals named; the probe records whether the space received whole conjunctions or single conjuncts | +| D-MTP-3 | recursive Datalog rules declared into `LanceSpace` → relalg `μ` through d′, queried by `match` over the derived relation | `anc_n` answers identical; no fixpoint state inside mask-risc | +| D-MTP-4 | the §3.1 expression-width panic as a fix on the fork, test first (separate from `1ffdbd6`) | red before, green after; upstream only with the operator's word | +| D-MTP-1b | the hop as a range (§6c): preimage of `valid`'s runs under a monotone `dst`, against the gather on clustered and scattered layouts | pairs == datafrog on F3/F4 and their scattered twins; two disable runs red; costs reported — **measured, §6d** | +| D-MTP-6 | datafrog as the exact Rust oracle in D-MTP-1/3 probes and the `ruff_metta_spo` lowerable tag (test-only dependency) | answers identical to Hyperon and PeTTa as bags of pairs, not only counts | +| D-MTP-5 | PeTTa as the second oracle in the probes (needs SWI-Prolog ≥ 9.3; 10.0.2 installed from the official PPA) | divergences between PeTTa and Hyperon recorded, never averaged | + +## 8. Fences + +- No second engine or interpreter inside lance-graph; no new opcode; no + MeTTa text past the binder. +- Hyperon's interpreter is not modified for this; the fork's upstream fixes + are separate PRs. +- An unsupported shape is a typed refusal answered by the in-memory space, + never an approximation. +- Licenses: Hyperon and PeTTa MIT; no Classic AtomSpace (AGPL) code. diff --git a/Cargo.toml b/Cargo.toml index f7b0de8d7..22d085929 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -115,6 +115,9 @@ exclude = [ # lance-graph-r2il like lance-graph-nal. Own workspace root; verify via # `cargo test --manifest-path crates/lance-graph-relalg/Cargo.toml`. "crates/lance-graph-relalg", + # D-MTP-1: its own workspace root (datafrog oracle is a git dev-dependency). + # `cargo test --manifest-path crates/lance-graph-metta/Cargo.toml`. + "crates/lance-graph-metta", "crates/symbiont", # OGAR (Open Graph of Active Record) activation crate — re-exports OGAR's full # AR surface (ogar-vocab Class/codebook + ogar-class-view impl ClassView + diff --git a/crates/lance-graph-metta/Cargo.toml b/crates/lance-graph-metta/Cargo.toml new file mode 100644 index 000000000..c2aa62b00 --- /dev/null +++ b/crates/lance-graph-metta/Cargo.toml @@ -0,0 +1,28 @@ +[package] +name = "lance-graph-metta" +version = "0.1.0" +edition = "2021" +publish = false +description = "MeTTa (Hyperon) as a declaration front end over the masking-algebra polyfill: the Datalog subset lowers through lance-graph-quack into mask-risc; everything else stays in Hyperon's interpreter. Plan: .claude/plans/metta-polyfill-v1.md." + +# Its OWN workspace root, like `lance-graph-relalg`: the datafrog oracle is a +# git dev-dependency that must not enter the main workspace's resolution. +# Verify via `cargo test --manifest-path crates/lance-graph-metta/Cargo.toml`. +[workspace] + +[dependencies] +# The only bulk engine, reached through Quack's lowering. +lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } +lance-graph-quack = { path = "../lance-graph-quack" } + +[dev-dependencies] +# D-MTP-6: the exact Datalog oracle, from the AdaWorldAPI fork. Probe/test only. +datafrog = { git = "https://github.com/AdaWorldAPI/datafrog", branch = "master" } + +[[example]] +name = "join_probe" +test = true + +[[example]] +name = "join_range_probe" +test = true diff --git a/crates/lance-graph-metta/examples/common/mod.rs b/crates/lance-graph-metta/examples/common/mod.rs new file mode 100644 index 000000000..8e98abf2f --- /dev/null +++ b/crates/lance-graph-metta/examples/common/mod.rs @@ -0,0 +1,254 @@ +//! Shared by the D-MTP probes: the fact layout, the gather path (D-MTP-1), +//! the range path (D-MTP-1b) and the datafrog oracle. Probe code, not a +//! shipped API. +#![allow(dead_code)] + +use std::time::Instant; + +use datafrog::Relation; +use lance_graph_mask_risc::exec::{execute_into, Scratch}; +use lance_graph_mask_risc::{words_for, Foreign, ForeignPlane, LaneRef, Out, Planes, Value}; +use lance_graph_quack::{lower, Agg, Col, Filter, ForeignPlane as QForeign, Mask, Query}; + +const DST: Col = Col(0); +const VALID: Mask = Mask(0); +const VALID_FOREIGN: QForeign = QForeign(0); + +/// A half-open row interval `[lo, hi)`. +pub type Run = (u32, u32); + +/// A functional binary relation laid out over its subjects' address space: +/// row `a` is the fact whose subject has address `a`, lane `dst` holds the +/// object's address, plane `valid` marks the addresses that have a fact. +pub struct Relation2 { + pub n: usize, + pub dst: Vec, + pub valid: Vec, + /// `valid`'s runs, recorded during ingest when the facts arrive in + /// address order (`None` otherwise). + pub declared_runs: Option>, + /// The order witness, checked once at ingest: `dst` non-decreasing. + pub monotone: bool, +} + +#[derive(Debug, PartialEq, Eq)] +pub enum IngestError { + /// A subject already has a fact: not a functional relation. + SecondFact { subject: u32 }, + /// An address outside the declared address space. + OutOfSpace { addr: u32 }, +} + +impl Relation2 { + /// Ingest facts `(subject, object)` over addresses `0..n`. A row without a + /// fact gets the previous row's `dst` (it lies outside `valid`, so the + /// value is never read as a fact; it only keeps a monotone lane monotone). + pub fn ingest(n: usize, facts: &[(u32, u32)]) -> Result { + let mut dst = vec![u32::MAX; n]; + let mut valid = vec![0u64; words_for(n)]; + let mut runs: Vec = Vec::new(); + let mut in_order = true; + let mut last: Option = None; + for &(s, o) in facts { + for a in [s, o] { + if a as usize >= n { + return Err(IngestError::OutOfSpace { addr: a }); + } + } + let (w, b) = (s as usize / 64, s as usize % 64); + if (valid[w] >> b) & 1 == 1 { + return Err(IngestError::SecondFact { subject: s }); + } + valid[w] |= 1 << b; + dst[s as usize] = o; + match (in_order, last) { + (true, Some(p)) if s <= p => in_order = false, + (true, _) => match runs.last_mut() { + Some(r) if r.1 == s => r.1 = s + 1, + _ => runs.push((s, s + 1)), + }, + _ => {} + } + last = Some(s); + } + let mut prev = 0u32; + for d in dst.iter_mut() { + if *d == u32::MAX { + *d = prev; + } else { + prev = *d; + } + } + let monotone = dst.windows(2).all(|w| w[0] <= w[1]); + Ok(Relation2 { + n, + dst, + valid, + declared_runs: in_order.then_some(runs), + monotone, + }) + } + + /// The order witness for the range path: `dst` non-decreasing over rows. + pub fn dst_monotone(&self) -> bool { + self.dst.windows(2).all(|w| w[0] <= w[1]) + } + + // ───────────── gather path (D-MTP-1): Plane(valid) AND Semijoin ───────────── + + fn query(&self) -> Filter { + Filter::and([Filter::plane(VALID), Filter::semijoin(DST, VALID_FOREIGN)]) + } + + fn run(&self, agg: Agg, kept: &mut [u64]) -> Value { + let keep = matches!(agg, Agg::Rows); + let program = lower(&Query { + filter: self.query(), + agg, + }) + .expect("lowers"); + let lanes = [LaneRef::U32(&self.dst)]; + let masks = [self.valid.as_slice()]; + let planes = Planes { + n_rows: self.n, + masks: &masks, + lanes: &lanes, + }; + let fplanes = [ForeignPlane { + words: &self.valid, + rows: self.n, + }]; + let foreign = Foreign { + planes: &fplanes, + lanes: &[], + }; + let mut scratch = Scratch::for_program(&program, self.n).expect("carves"); + let out = if keep { Out::Mask(kept) } else { Out::None }; + execute_into(&program, &planes, &foreign, &mut scratch, out).expect("runs") + } + + /// `size-atom` of the collapsed result, by the gather program. + pub fn count(&self) -> usize { + match self.run(Agg::Count, &mut []) { + Value::Count(c) => c, + other => panic!("not a count: {other:?}"), + } + } + + /// The bindings `($x, $z)`, emitted only for rows the gather program kept. + pub fn pairs(&self, kept: &mut [u64], out: &mut Vec<(u32, u32)>) { + match self.run(Agg::Rows, kept) { + Value::Mask(_) => {} + other => panic!("not a kept mask: {other:?}"), + } + out.clear(); + for (w, &word) in kept.iter().enumerate() { + let mut bits = word; + while bits != 0 { + let x = (w * 64 + bits.trailing_zeros() as usize) as u32; + let y = self.dst[x as usize]; + out.push((x, self.dst[y as usize])); + bits &= bits - 1; + } + } + } + + // ───────────── range path (D-MTP-1b): the hop as interval arithmetic ───────────── + + /// `valid`'s runs, found by a word scan of the plane. + pub fn scanned_runs(&self) -> Vec { + let mut runs: Vec = Vec::new(); + for (w, &word) in self.valid.iter().enumerate() { + let mut bits = word; + let base = (w * 64) as u32; + while bits != 0 { + let start = bits.trailing_zeros(); + let len = (!(bits >> start)).trailing_zeros().min(64 - start); + let (lo, hi) = (base + start, base + start + len); + match runs.last_mut() { + Some(r) if r.1 == lo => r.1 = hi, + _ => runs.push((lo, hi)), + } + bits = if start + len >= 64 { 0 } else { bits & (u64::MAX << (start + len)) }; + } + } + runs + } + + /// The rows whose target lies in one of `runs`: one interval per run, + /// found by binary search on the monotone `dst` lane. + pub fn preimage(&self, runs: &[Run]) -> Vec { + runs.iter() + .filter_map(|&(a, b)| { + let lo = self.dst.partition_point(|&d| d < a) as u32; + let hi = self.dst.partition_point(|&d| d < b) as u32; + (lo < hi).then_some((lo, hi)) + }) + .collect() + } + + /// The kept rows of `Plane(valid) AND Semijoin(dst, valid)` as intervals. + /// `None` when the order witness fails: the caller uses the gather. + pub fn kept_ranges(&self, valid_runs: &[Run]) -> Option> { + if !self.monotone { + return None; + } + Some(intersect(valid_runs, &self.preimage(valid_runs))) + } + + /// The bindings `($x, $z)` by walking the kept intervals. + pub fn range_pairs(&self, kept: &[Run], out: &mut Vec<(u32, u32)>) { + out.clear(); + for &(lo, hi) in kept { + for x in lo..hi { + let y = self.dst[x as usize]; + out.push((x, self.dst[y as usize])); + } + } + } +} + +/// Intersection of two sorted, disjoint interval lists. +pub fn intersect(a: &[Run], b: &[Run]) -> Vec { + let (mut i, mut j) = (0, 0); + let mut out = Vec::new(); + while i < a.len() && j < b.len() { + let lo = a[i].0.max(b[j].0); + let hi = a[i].1.min(b[j].1); + if lo < hi { + out.push((lo, hi)); + } + if a[i].1 < b[j].1 { + i += 1; + } else { + j += 1; + } + } + out +} + +/// Total length of an interval list. +pub fn total(runs: &[Run]) -> usize { + runs.iter().map(|&(lo, hi)| (hi - lo) as usize).sum() +} + +/// The datafrog oracle: `(x, z)` with `parent(x, y)` and `parent(y, z)`. +pub fn oracle(facts: &[(u32, u32)]) -> Vec<(u32, u32)> { + let parent: Relation<(u32, u32)> = facts.iter().copied().collect(); + let by_dst: Relation<(u32, u32)> = facts.iter().map(|&(x, y)| (y, x)).collect(); + let pairs = Relation::from_join(&by_dst, &parent, |_y, &x, &z| (x, z)); + pairs.elements.clone() +} + +/// Best wall time of `reps` runs, and the last result. +pub fn best(reps: usize, mut f: impl FnMut() -> T) -> (T, f64) { + let mut best = f64::INFINITY; + let mut out = None; + for _ in 0..reps { + let t = Instant::now(); + let r = f(); + best = best.min(t.elapsed().as_secs_f64()); + out = Some(r); + } + (out.expect("reps > 0"), best) +} diff --git a/crates/lance-graph-metta/examples/join_probe.rs b/crates/lance-graph-metta/examples/join_probe.rs new file mode 100644 index 000000000..6ea973de3 --- /dev/null +++ b/crates/lance-graph-metta/examples/join_probe.rs @@ -0,0 +1,276 @@ +//! D-MTP-1 — the two-hop MeTTa join lowered through Quack into mask-risc. +//! +//! MeTTa: `(match &kb (, (parent $x $y) (parent $y $z)) ($x $z))`. +//! +//! Layout (ingest): a functional binary relation over the subject's address +//! space. Row `a` is the fact whose subject has address `a`; lane `dst` holds +//! the object's address; plane `valid` marks the addresses that have a fact. +//! A second fact for the same subject is refused. +//! +//! Lowering: `Plane(valid) AND Semijoin { fk: dst, foreign: valid }`, then +//! `Count` (the size of the collapsed result) or `Rows` + binding emission +//! `(x, dst[dst[x]])` over the kept bits — the only place pairs exist. +//! +//! Oracle: datafrog (AdaWorldAPI fork), compared as sorted pair vectors. +//! Pre-registration: plan §6a. +//! +//! ```text +//! CARGO_PROFILE_RELEASE_DEBUG=0 RUSTFLAGS=-Ctarget-cpu=native \ +//! cargo run --release --manifest-path crates/lance-graph-metta/Cargo.toml --example join_probe +//! ``` + +use std::time::Instant; + +use datafrog::Relation; +use lance_graph_mask_risc::exec::{execute_into, Scratch}; +use lance_graph_mask_risc::{words_for, Foreign, ForeignPlane, LaneRef, Out, Planes, Value}; +use lance_graph_quack::{lower, Agg, Col, Filter, ForeignPlane as QForeign, Mask, Query}; + +const DST: Col = Col(0); +const VALID: Mask = Mask(0); +const VALID_FOREIGN: QForeign = QForeign(0); + +/// A functional binary relation laid out over its subjects' address space. +struct Relation2 { + n: usize, + dst: Vec, + valid: Vec, +} + +#[derive(Debug, PartialEq, Eq)] +enum IngestError { + /// A subject already has a fact: not a functional relation. + SecondFact { subject: u32 }, + /// An address outside the declared address space. + OutOfSpace { addr: u32 }, +} + +impl Relation2 { + /// Ingest facts `(subject, object)` over addresses `0..n`. + fn ingest(n: usize, facts: &[(u32, u32)]) -> Result { + let mut dst = vec![0u32; n]; + let mut valid = vec![0u64; words_for(n)]; + for &(s, o) in facts { + for a in [s, o] { + if a as usize >= n { + return Err(IngestError::OutOfSpace { addr: a }); + } + } + let (w, b) = (s as usize / 64, s as usize % 64); + if (valid[w] >> b) & 1 == 1 { + return Err(IngestError::SecondFact { subject: s }); + } + valid[w] |= 1 << b; + dst[s as usize] = o; + } + Ok(Relation2 { n, dst, valid }) + } + + fn query(&self) -> Filter { + Filter::and([Filter::plane(VALID), Filter::semijoin(DST, VALID_FOREIGN)]) + } + + fn run(&self, agg: Agg, kept: &mut [u64]) -> Value { + let keep = matches!(agg, Agg::Rows); + let program = lower(&Query { + filter: self.query(), + agg, + }) + .expect("lowers"); + let lanes = [LaneRef::U32(&self.dst)]; + let masks = [self.valid.as_slice()]; + let planes = Planes { + n_rows: self.n, + masks: &masks, + lanes: &lanes, + }; + let fplanes = [ForeignPlane { + words: &self.valid, + rows: self.n, + }]; + let foreign = Foreign { + planes: &fplanes, + lanes: &[], + }; + let mut scratch = Scratch::for_program(&program, self.n).expect("carves"); + let out = if keep { Out::Mask(kept) } else { Out::None }; + execute_into(&program, &planes, &foreign, &mut scratch, out).expect("runs") + } + + /// `size-atom` of the collapsed result. + fn count(&self) -> usize { + match self.run(Agg::Count, &mut []) { + Value::Count(c) => c, + other => panic!("not a count: {other:?}"), + } + } + + /// The bindings `($x, $z)`, emitted only for kept rows. + fn pairs(&self, kept: &mut [u64], out: &mut Vec<(u32, u32)>) { + match self.run(Agg::Rows, kept) { + Value::Mask(_) => {} + other => panic!("not a kept mask: {other:?}"), + } + out.clear(); + for (w, &word) in kept.iter().enumerate() { + let mut bits = word; + while bits != 0 { + let x = (w * 64 + bits.trailing_zeros() as usize) as u32; + let y = self.dst[x as usize]; + out.push((x, self.dst[y as usize])); + bits &= bits - 1; + } + } + } +} + +/// F1 (`partial = false`): `parent(i, (7i+3) mod n)` for every `i`. +/// F2 (`partial = true`): the same, only for `i mod 3 != 0`. +fn facts(n: u32, partial: bool) -> Vec<(u32, u32)> { + (0..n) + .filter(|i| !partial || i % 3 != 0) + .map(|i| (i, ((7u64 * i as u64 + 3) % n as u64) as u32)) + .collect() +} + +/// The datafrog oracle: `(x, z)` with `parent(x, y)` and `parent(y, z)`. +fn oracle(facts: &[(u32, u32)]) -> Vec<(u32, u32)> { + let parent: Relation<(u32, u32)> = facts.iter().copied().collect(); + let by_dst: Relation<(u32, u32)> = facts.iter().map(|&(x, y)| (y, x)).collect(); + let pairs = Relation::from_join(&by_dst, &parent, |_y, &x, &z| (x, z)); + pairs.elements.clone() +} + +fn best(reps: usize, mut f: impl FnMut() -> T) -> (T, f64) { + let mut best = f64::INFINITY; + let mut out = None; + for _ in 0..reps { + let t = Instant::now(); + let r = f(); + best = best.min(t.elapsed().as_secs_f64()); + out = Some(r); + } + (out.expect("reps > 0"), best) +} + +fn main() { + println!( + "tier: avx512f={} avx2={}", + cfg!(target_feature = "avx512f"), + cfg!(target_feature = "avx2") + ); + println!( + "fixture n | pairs | ingest ns/fact | count ns/row | rows+emit ns/pair | datafrog ns/pair" + ); + for partial in [false, true] { + for n in [100u32, 1_000, 10_000, 100_000, 1_000_000] { + let f = facts(n, partial); + let (rel, t_ingest) = + best(5, || Relation2::ingest(n as usize, &f).expect("functional")); + let (count, t_count) = best(5, || rel.count()); + let mut kept = vec![0u64; words_for(n as usize)]; + let mut pairs = Vec::with_capacity(n as usize); + let (_, t_pairs) = best(5, || rel.pairs(&mut kept, &mut pairs)); + let (want, t_frog) = best(5, || oracle(&f)); + let mut got = pairs.clone(); + got.sort_unstable(); + assert_eq!( + got, + want, + "pairs differ from datafrog (F{}, n={n})", + 1 + partial as u8 + ); + assert_eq!(count, want.len(), "count differs from datafrog"); + let p = want.len().max(1) as f64; + println!( + "F{} {n:>9} | {:>7} | {:>6.2} | {:>6.3} | {:>6.2} | {:>6.1}", + 1 + partial as u8, + want.len(), + t_ingest * 1e9 / f.len() as f64, + t_count * 1e9 / n as f64, + t_pairs * 1e9 / p, + t_frog * 1e9 / p, + ); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn check(n: u32, partial: bool) { + let f = facts(n, partial); + let rel = Relation2::ingest(n as usize, &f).expect("functional"); + let mut kept = vec![0u64; words_for(n as usize)]; + let mut got = Vec::new(); + rel.pairs(&mut kept, &mut got); + got.sort_unstable(); + let want = oracle(&f); + assert_eq!(got, want, "F{} n={n}", 1 + partial as u8); + assert_eq!(rel.count(), want.len(), "F{} n={n}", 1 + partial as u8); + } + + #[test] + fn f1_total_relation_matches_datafrog() { + for n in [1, 2, 63, 64, 65, 100, 1_000] { + check(n, false); + } + } + + #[test] + fn f2_partial_relation_matches_datafrog() { + for n in [3, 63, 64, 65, 100, 1_000] { + check(n, true); + } + } + + /// Anti-vacuity: F2 must reject rows, or it cannot tell a semijoin from + /// no semijoin. + #[test] + fn f2_semijoin_rejects_rows() { + let f = facts(1_000, true); + let want = oracle(&f); + assert!( + want.len() < f.len(), + "{} of {} facts survive", + want.len(), + f.len() + ); + assert!( + want.len() * 2 > f.len(), + "fixture too sparse to be informative" + ); + } + + /// The counts Hyperon and PeTTa returned for the same MeTTa programs. + #[test] + fn counts_match_hyperon_and_petta() { + // F1 from the D-MTP-0 runs; F2 measured 2026-10-14: Hyperon 46 / 476, + // PeTTa 46 / 476 / 4,762. + for (n, partial, expected) in [ + (100, false, 100), + (1_000, false, 1_000), + (100, true, 46), + (1_000, true, 476), + (10_000, true, 4_762), + ] { + let f = facts(n, partial); + let rel = Relation2::ingest(n as usize, &f).expect("functional"); + assert_eq!(rel.count(), expected, "n={n}"); + } + } + + #[test] + fn ingest_refuses_a_second_fact_for_one_subject() { + assert_eq!( + Relation2::ingest(4, &[(1, 2), (1, 3)]).err(), + Some(IngestError::SecondFact { subject: 1 }) + ); + assert_eq!( + Relation2::ingest(4, &[(1, 4)]).err(), + Some(IngestError::OutOfSpace { addr: 4 }) + ); + assert!(Relation2::ingest(4, &[(1, 2), (2, 1)]).is_ok()); + } +} diff --git a/crates/lance-graph-metta/examples/join_range_probe.rs b/crates/lance-graph-metta/examples/join_range_probe.rs new file mode 100644 index 000000000..2c549d30e --- /dev/null +++ b/crates/lance-graph-metta/examples/join_range_probe.rs @@ -0,0 +1,203 @@ +//! D-MTP-1b — the hop as a range (plan §6c). +//! +//! MeTTa: `(match &kb (, (parent $x $y) (parent $y $z)) ($x $z))`, the same +//! two-hop join as D-MTP-1. When the link lane `dst` is non-decreasing over +//! rows, `Semijoin(dst, valid)` restricted to `valid` is an intersection of +//! interval sets: `valid`'s runs and their preimages under `dst` (found by +//! `partition_point`). The count is interval arithmetic; pairs are emitted by +//! walking the kept intervals. A non-monotone lane is refused and the caller +//! keeps the gather program. +//! +//! Arms: G-scattered / G-clustered (the D-MTP-1 gather program), R-scan +//! (`valid`'s runs from a word scan), R-declared (runs recorded at ingest). +//! Oracle: datafrog (AdaWorldAPI fork). +//! +//! ```text +//! CARGO_PROFILE_RELEASE_DEBUG=0 RUSTFLAGS=-Ctarget-cpu=native \ +//! cargo run --release --manifest-path crates/lance-graph-metta/Cargo.toml --example join_range_probe +//! ``` + +mod common; + +use common::{best, oracle, total, Relation2}; + +/// Depth of node `x` in the fanout-4 BFS tree (`parent(x) = (x-1)/4`). +fn depth(mut x: u32) -> u32 { + let mut d = 0; + while x > 0 { + x = (x - 1) / 4; + d += 1; + } + d +} + +/// F3 (`banded = false`) or F4 (`banded = true`: no facts for nodes whose +/// depth is a multiple of 3), in address order. +fn tree(n: u32, banded: bool) -> Vec<(u32, u32)> { + (1..n) + .filter(|&x| !banded || depth(x) % 3 != 0) + .map(|x| (x, (x - 1) / 4)) + .collect() +} + +/// A deterministic random permutation of `0..n` (xorshift + Fisher–Yates). +fn permutation(n: u32) -> Vec { + let mut p: Vec = (0..n).collect(); + let mut s: u64 = 0x9E37_79B9_7F4A_7C15; + for i in (1..n as usize).rev() { + s ^= s << 13; + s ^= s >> 7; + s ^= s << 17; + p.swap(i, (s % (i as u64 + 1)) as usize); + } + p +} + +/// The same facts under a permutation of addresses (the scattered twin). +fn scatter(n: u32, facts: &[(u32, u32)]) -> Vec<(u32, u32)> { + let p = permutation(n); + facts.iter().map(|&(x, y)| (p[x as usize], p[y as usize])).collect() +} + +fn sorted(mut v: Vec<(u32, u32)>) -> Vec<(u32, u32)> { + v.sort_unstable(); + v +} + +#[cfg(test)] +mod tests { + use super::*; + use common::intersect; + + const SIZES: [u32; 3] = [100, 1000, 4097]; + + fn gather_pairs(r: &Relation2) -> Vec<(u32, u32)> { + let mut kept = vec![0u64; r.valid.len()]; + let mut out = Vec::new(); + r.pairs(&mut kept, &mut out); + sorted(out) + } + + fn range_pairs(r: &Relation2, runs: &[(u32, u32)]) -> Vec<(u32, u32)> { + let kept = r.kept_ranges(runs).expect("monotone"); + let mut out = Vec::new(); + r.range_pairs(&kept, &mut out); + sorted(out) + } + + #[test] + fn every_arm_equals_datafrog() { + for n in SIZES { + for banded in [false, true] { + let f = tree(n, banded); + let want = sorted(oracle(&f)); + assert!(!want.is_empty()); + let r = Relation2::ingest(n as usize, &f).unwrap(); + assert!(r.dst_monotone() && r.monotone); + let declared = r.declared_runs.clone().expect("address order"); + let scanned = r.scanned_runs(); + assert_eq!(declared, scanned, "n={n} banded={banded}"); + assert_eq!(gather_pairs(&r), want, "G-clustered n={n}"); + assert_eq!(range_pairs(&r, &scanned), want, "R n={n} banded={banded}"); + assert_eq!(r.count(), want.len()); + let kept = r.kept_ranges(&scanned).unwrap(); + assert_eq!(total(&kept), want.len()); + + let s = scatter(n, &f); + let rs = Relation2::ingest(n as usize, &s).unwrap(); + assert_eq!(gather_pairs(&rs).len(), want.len(), "G-scattered n={n}"); + assert_eq!(sorted(oracle(&s)), gather_pairs(&rs)); + } + } + } + + #[test] + fn a_non_monotone_lane_is_refused() { + let s = scatter(1000, &tree(1000, true)); + let r = Relation2::ingest(1000, &s).unwrap(); + assert!(!r.dst_monotone() && !r.monotone); + assert!(r.declared_runs.is_none()); + assert!(r.kept_ranges(&r.scanned_runs()).is_none()); + } + + /// Both halves of the intersection must matter on F4, or the disable runs + /// in plan §6c prove nothing. + #[test] + fn f4_needs_both_halves_of_the_intersection() { + let n = 4097; + let r = Relation2::ingest(n as usize, &tree(n, true)).unwrap(); + let runs = r.scanned_runs(); + // Levels 1-2 and 4-5 merge: [1, 21) and [85, 1365) at n = 4097. + assert_eq!(runs, vec![(1, 21), (85, 1365)], "F4 valid runs"); + let want = total(&r.kept_ranges(&runs).unwrap()); + assert_ne!(total(&r.preimage(&runs)), want, "preimage alone"); + assert_ne!(total(&runs), want, "valid alone"); + // F3: one run, and the preimage alone already equals the answer. + let r3 = Relation2::ingest(n as usize, &tree(n, false)).unwrap(); + let runs3 = r3.scanned_runs(); + assert_eq!(runs3.len(), 1); + assert_eq!( + intersect(&runs3, &r3.preimage(&runs3)), + r3.kept_ranges(&runs3).unwrap() + ); + } +} + +fn main() { + println!( + "tier: avx512f={} avx2={}", + cfg!(target_feature = "avx512f"), + cfg!(target_feature = "avx2") + ); + const REPS: usize = 5; + for n in [1_000u32, 10_000, 100_000, 1_000_000] { + for (name, banded) in [("F3", false), ("F4", true)] { + let f = tree(n, banded); + let s = scatter(n, &f); + let (want, t_oracle) = best(REPS, || oracle(&f)); + let want = sorted(want); + let pairs = want.len(); + + let (r, t_ingest) = best(REPS, || Relation2::ingest(n as usize, &f).unwrap()); + let (rs, t_ingest_s) = best(REPS, || Relation2::ingest(n as usize, &s).unwrap()); + let (_, t_mono) = best(REPS, || std::hint::black_box(std::hint::black_box(&r).dst_monotone())); + + let (c, t_gc) = best(REPS, || r.count()); + let (cs, t_gs) = best(REPS, || rs.count()); + let mut kept = vec![0u64; r.valid.len()]; + let mut out = Vec::with_capacity(pairs); + let (_, t_gc_pairs) = best(REPS, || r.pairs(&mut kept, &mut out)); + assert_eq!(sorted(out.clone()), want); + let (_, t_gs_pairs) = best(REPS, || rs.pairs(&mut kept, &mut out)); + assert_eq!(out.len(), pairs); + + let (rc, t_rscan) = best(REPS, || total(&r.kept_ranges(&r.scanned_runs()).unwrap())); + let declared = r.declared_runs.clone().unwrap(); + let (rd, t_rdecl) = best(REPS, || total(&r.kept_ranges(&declared).unwrap())); + let kept_r = r.kept_ranges(&declared).unwrap(); + let (_, t_r_pairs) = best(REPS, || r.range_pairs(&kept_r, &mut out)); + assert_eq!(sorted(out.clone()), want); + assert!(c == pairs && cs == pairs && rc == pairs && rd == pairs); + + let per_row = |t: f64| t * 1e9 / n as f64; + let per_pair = |t: f64| t * 1e9 / pairs.max(1) as f64; + println!( + "{name} n={n:>8} pairs={pairs:>8} runs={:>3} | count ns/row: G-scat {:.3} G-clus {:.3} R-scan {:.4} R-decl {:.5} \ + | emit ns/pair: G-scat {:.2} G-clus {:.2} R {:.2} | ingest ns/row: clus {:.2} scat {:.2} mono {:.3} \ + | datafrog ns/pair {:.1}", + declared.len(), + per_row(t_gs), + per_row(t_gc), + per_row(t_rscan), + per_row(t_rdecl), + per_pair(t_gs_pairs), + per_pair(t_gc_pairs), + per_pair(t_r_pairs), + per_row(t_ingest), + per_row(t_ingest_s), + per_row(t_mono), + per_pair(t_oracle), + ); + } + } +} diff --git a/crates/lance-graph-metta/src/lib.rs b/crates/lance-graph-metta/src/lib.rs new file mode 100644 index 000000000..acb39a333 --- /dev/null +++ b/crates/lance-graph-metta/src/lib.rs @@ -0,0 +1,7 @@ +//! MeTTa (Hyperon) as a declaration front end over the masking-algebra +//! polyfill. Nothing is shipped here yet: D-MTP-1 is a probe +//! (`examples/join_probe.rs`) that lowers one MeTTa conjunctive query by hand +//! through `lance-graph-quack` into one mask-risc program and checks it +//! against the datafrog oracle. The `Space` adapter is D-MTP-2. +//! +//! Plan: `.claude/plans/metta-polyfill-v1.md`.