Skip to content

Commit 698c1c3

Browse files
committed
feat(hpc): clam_v3 — V3-Distanzen fuer CLAM, Region statt Ganz-Zeile
ClamTree::build defaultet auf hamming_inline ueber den GANZEN Punkt. Fuer flache Fingerprints richtig; fuer eine V3-SoA-Zeile ein Kategorienfehler: key(16)|edges(16)|value(480) sind drei Regionen mit drei Bedeutungen, und der Key ist eine ADRESSE. Hamming zwischen Adressen clustert auf Adress-Rauschen — und sieht plausibel aus, weil jede Zahl wohlgeformt ist. Zwei Distanzen, jede liest genau eine Region: V3RailGeodesic Rail-Register Baum-Geodaete d(a,b)=da+db-2*lca v3_value_hamming 32.. Inhalt; DistanceFn-kompatibel Das Facet-Payload wird als 6x(u8:u8) gelesen — sechs Ebenen, je ein Achsen-PAAR (X:Y, z.B. part_of:is_a), zwei getrennte Bytes, niemals zu u16 verbreitert (ein verbreitertes Wort hat keine Achse; genau deshalb ist der flache Tail V1-legacy). Der erste Entwurf dieser Datei las 3xu16 HEEL/HIP/TWIG — die zurueckgezogene V1-Kaskaden-Lesart; vom Operator mitten im Schreiben gefangen. Die LESART gehoert der ClassView, also ist sie hier ein PARAMETER: RailSpec (Register-Offset, Achse als Enum, optionales gestapeltes Fortsetzungs-Register z.B. in der Edge-Lane) wird vom Aufrufer aus seiner ClassView/WideFieldMask abgeleitet. ndarray loest niemals eine classid auf — es weigert sich nur zu raten. v3_facet() ist der Zero-Fallback-Default und sagt das dazu. Stapeln statt verbreitern: stacked() verkettet ein zweites Register, Tiefe 0..=12, gleiche Loch-Regel. Grenzen als Tests gepinnt statt als Fussnoten: Dreiecksungleichung (was CLAMs Pruning wirklich braucht) ueber eine geschlossene Menge; die Pseudometrik-Grenze (zwei echt verschiedene Knoten jenseits der gespeicherten Tiefe messen 0) als BENANNTER gruener Test; Lo sieht Hi nicht (der Test, den ein u16 nicht bestehen koennte); Wert-Distanz liest die Adresse nie. Keine Distanz ueber den Edge-Block ohne aufgeloesten edge_codec_flavor — verweigert, nicht geraten.
1 parent 6ff231a commit 698c1c3

2 files changed

Lines changed: 379 additions & 0 deletions

File tree

‎src/hpc/clam_v3.rs‎

Lines changed: 378 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,378 @@
1+
//! V3 distances for [`super::clam`] — read the row's REGIONS, never the whole row.
2+
//!
3+
//! # Why the default is wrong for a V3 row
4+
//!
5+
//! `ClamTree::build` / `build_with_config` route to `hamming_inline`, which
6+
//! Hammings the **whole** point. Correct for flat fingerprints; a category
7+
//! error for a V3 SoA row, which is three regions with three meanings:
8+
//!
9+
//! ```text
10+
//! 512-byte V3 row
11+
//! 0..16 key classid(4) | facet payload(12)
12+
//! 16..32 edges EdgeBlock — reading is per-class (edge_codec_flavor)
13+
//! 32..512 value tenant slab
14+
//! ```
15+
//!
16+
//! **The key is an ADDRESS.** Hamming between addresses clusters on address
17+
//! noise, and it looks plausible because every number is well-formed. That is
18+
//! the same defect as putting a count on the wire as a radius — a value read
19+
//! outside the register that gives it meaning.
20+
//!
21+
//! # The facet register: 6×(u8:u8), axis-paired — NOT 3×u16
22+
//!
23+
//! The 12-byte payload is a content-blind register the classid's ClassView
24+
//! projects. Its rail carving is **six levels of an axis PAIR** —
25+
//! `X:Y` per level, e.g. `part_of:is_a` — two separate bytes, **never
26+
//! widened to u16** (the flat-u24/u16 tail is the retired V1 shape: a widened
27+
//! word has no axis and cannot carry a rail). A byte value is `1 + index`;
28+
//! `0` means *no such level*.
29+
//!
30+
//! So the tree read is: pick ONE axis of the pair, walk the six levels,
31+
//! stop at the first zero. Depth is a count, LCA is a leading-agreement
32+
//! count, and the tree geodesic
33+
//!
34+
//! ```text
35+
//! d(a,b) = depth(a) + depth(b) - 2·depth(lca(a,b))
36+
//! ```
37+
//!
38+
//! costs a handful of byte compares. Position IS the information: byte i of
39+
//! the walked axis is level i, and which bytes are live is a field-mask
40+
//! question, not a parsing question.
41+
//!
42+
//! # The reading comes from the ClassView — so it is a PARAMETER here
43+
//!
44+
//! ndarray is the foundation crate; it cannot (and must not) resolve a
45+
//! classid to its ClassView. What it can do is refuse to hardcode one
46+
//! reading: [`RailSpec`] is the byte-range + axis handle a caller derives
47+
//! from its ClassView / WideFieldMask and passes in. The unconfigured
48+
//! convenience ([`RailSpec::v3_facet`]) is only the zero-fallback default —
49+
//! primary register at `4..16`, axis 0, no continuation.
50+
//!
51+
//! **Stacking:** a class that needs more than six levels does not widen a
52+
//! byte — it stacks a second register, e.g. into the edge lane, and chains
53+
//! it (`RailSpec::stacked`). Depth then runs 0..=12 over two registers, same
54+
//! hole rule, same arithmetic. Continuation is a spec decision made where
55+
//! the ClassView lives, never assumed here.
56+
//!
57+
//! # Honest limits
58+
//!
59+
//! 1. **Bounded depth ⇒ pseudometric.** Past the stored levels, distinct
60+
//! nodes with an identical walked prefix measure 0 apart. Triangle
61+
//! inequality holds (CLAM pruning stays sound); identity of
62+
//! indiscernibles does not.
63+
//! 2. **A DAG is not a tree.** The rail linearises through one chosen
64+
//! parent; pairs related only through a non-chosen parent read far. The
65+
//! linearisation question belongs to the bake, not to this function.
66+
//! 3. **No distance over the edge block without its flavor.** `16..32` may
67+
//! be a continuation register (if the spec says so) — but a *similarity*
68+
//! over edge bytes needs `edge_codec_flavor` resolved first and is
69+
//! deliberately absent here.
70+
71+
use super::clam::Distance;
72+
73+
/// V3 key region: `classid(4) | facet payload(12)`.
74+
pub const V3_KEY_LEN: usize = 16;
75+
/// V3 edge block, following the key.
76+
pub const V3_EDGE_LEN: usize = 16;
77+
/// First byte a CONTENT distance may read.
78+
pub const V3_VALUE_OFF: usize = V3_KEY_LEN + V3_EDGE_LEN;
79+
/// Levels per rail register: six (u8:u8) pairs in the 12-byte facet payload.
80+
pub const RAIL_PAIRS: usize = 6;
81+
82+
/// Which byte of each `(u8:u8)` level pair a walk reads.
83+
///
84+
/// The pair is an axis pair (`X:Y`, e.g. `part_of:is_a`) — two SEPARATE
85+
/// bytes. An enum rather than a `u8` so a third axis cannot be conjured by
86+
/// arithmetic: the pair has two lanes, and that is a property of the carving,
87+
/// not a parameter.
88+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
89+
pub enum RailAxis {
90+
/// Byte 0 of each pair (the `X` of `X:Y`).
91+
Lo,
92+
/// Byte 1 of each pair (the `Y` of `X:Y`).
93+
Hi,
94+
}
95+
96+
/// The byte-range + axis handle for reading a row's rail register(s).
97+
///
98+
/// This is the ClassView's reading made portable: the caller resolves WHICH
99+
/// bytes carry the register and which axis is walked from its
100+
/// ClassView / WideFieldMask, and hands the result here. ndarray never
101+
/// resolves a classid — it only refuses to guess.
102+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
103+
pub struct RailSpec {
104+
/// Byte offset of the primary register (6 pairs = 12 bytes).
105+
pub reg: usize,
106+
/// Optional stacked continuation register (6 more pairs), e.g. carved
107+
/// into the edge lane when the ClassView says so. `None` = 6 levels max.
108+
pub cont: Option<usize>,
109+
/// Which byte of each level pair the walk reads.
110+
pub axis: RailAxis,
111+
}
112+
113+
impl RailSpec {
114+
/// The zero-fallback default: primary register at `4..16` (the facet
115+
/// payload), axis `Lo`, no continuation. Correct for a class whose
116+
/// ClassView has not said otherwise — and ONLY for such a class.
117+
#[must_use]
118+
pub const fn v3_facet() -> Self {
119+
Self { reg: 4, cont: None, axis: RailAxis::Lo }
120+
}
121+
122+
/// The same spec with a continuation register stacked at `at` —
123+
/// e.g. `V3_KEY_LEN` when the ClassView carves it into the edge lane.
124+
#[must_use]
125+
pub const fn stacked(self, at: usize) -> Self {
126+
Self { cont: Some(at), ..self }
127+
}
128+
129+
/// Maximum representable depth under this spec.
130+
#[must_use]
131+
pub const fn max_depth(&self) -> u32 {
132+
if self.cont.is_some() { 2 * RAIL_PAIRS as u32 } else { RAIL_PAIRS as u32 }
133+
}
134+
135+
/// The walked-axis byte of level `i` (0-based), or 0 if out of range.
136+
/// Position is the information: byte index IS the level.
137+
#[inline]
138+
fn level(&self, row: &[u8], i: usize) -> u8 {
139+
let (base, k) = if i < RAIL_PAIRS {
140+
(self.reg, i)
141+
} else {
142+
match self.cont {
143+
Some(c) => (c, i - RAIL_PAIRS),
144+
None => return 0,
145+
}
146+
};
147+
let at = base + 2 * k + match self.axis { RailAxis::Lo => 0, RailAxis::Hi => 1 };
148+
if at < row.len() { row[at] } else { 0 }
149+
}
150+
151+
/// Depth = occupied leading levels. Stops at the first zero: a hole ends
152+
/// the chain, and a value after a hole is not ancestry (`[1,0,7]` is
153+
/// depth 1, never 2).
154+
#[must_use]
155+
pub fn depth(&self, row: &[u8]) -> u32 {
156+
let mut d = 0;
157+
for i in 0..self.max_depth() as usize {
158+
if self.level(row, i) == 0 {
159+
break;
160+
}
161+
d += 1;
162+
}
163+
d
164+
}
165+
166+
/// Depth of the lowest common ancestor: leading levels that are occupied
167+
/// and equal on the walked axis.
168+
#[must_use]
169+
pub fn lca_depth(&self, a: &[u8], b: &[u8]) -> u32 {
170+
let mut d = 0;
171+
for i in 0..self.max_depth() as usize {
172+
let (x, y) = (self.level(a, i), self.level(b, i));
173+
if x == 0 || x != y {
174+
break;
175+
}
176+
d += 1;
177+
}
178+
d
179+
}
180+
181+
/// The tree geodesic on the linearisation:
182+
/// `depth(a) + depth(b) - 2·lca`.
183+
#[must_use]
184+
pub fn geodesic(&self, a: &[u8], b: &[u8]) -> u64 {
185+
let (da, db, l) = (self.depth(a), self.depth(b), self.lca_depth(a, b));
186+
u64::from(da - l) + u64::from(db - l)
187+
}
188+
}
189+
190+
/// [`Distance`] over the rail register — the TREE distance, ClassView-shaped.
191+
///
192+
/// `is_metric()` returns `true` because CLAM's pruning only needs the
193+
/// triangle inequality, which the tree geodesic satisfies. Read the module
194+
/// doc before trusting it further: bounded depth makes this a
195+
/// **pseudometric** (distinct deep nodes can measure 0), and on a DAG it
196+
/// measures the linearisation, not the full relation.
197+
#[derive(Debug, Clone, Copy)]
198+
pub struct V3RailGeodesic(pub RailSpec);
199+
200+
impl Distance for V3RailGeodesic {
201+
type Point = [u8];
202+
fn distance(&self, a: &[u8], b: &[u8]) -> u64 {
203+
self.0.geodesic(a, b)
204+
}
205+
fn is_metric(&self) -> bool {
206+
true
207+
}
208+
}
209+
210+
/// [`Distance`] over the value slab only — the CONTENT distance.
211+
///
212+
/// Reads `32..` and nothing else: two rows with identical tenants and
213+
/// different addresses are distance 0, which is the point — the address is
214+
/// where a thing lives, not what it is.
215+
#[derive(Debug, Clone, Copy, Default)]
216+
pub struct V3ValueHamming;
217+
218+
impl Distance for V3ValueHamming {
219+
type Point = [u8];
220+
fn distance(&self, a: &[u8], b: &[u8]) -> u64 {
221+
v3_value_hamming(a, b)
222+
}
223+
fn is_metric(&self) -> bool {
224+
true
225+
}
226+
}
227+
228+
/// The content distance as a bare fn, signature-compatible with
229+
/// [`super::clam::DistanceFn`] so it plugs straight into
230+
/// `ClamTree::build_with_fn(rows, 512, …, v3_value_hamming)`.
231+
///
232+
/// A row shorter than the value offset contributes nothing — an honest 0
233+
/// beats a panic in a distance callback, and a truncated row is a loader
234+
/// bug this function cannot repair.
235+
#[must_use]
236+
pub fn v3_value_hamming(a: &[u8], b: &[u8]) -> u64 {
237+
if a.len() <= V3_VALUE_OFF || b.len() <= V3_VALUE_OFF {
238+
return 0;
239+
}
240+
super::clam::hamming_inline(&a[V3_VALUE_OFF..], &b[V3_VALUE_OFF..])
241+
}
242+
243+
#[cfg(test)]
244+
mod tests {
245+
use super::*;
246+
247+
/// A 512-byte row with the given rail levels written on the chosen axis
248+
/// of the facet payload, and a value slab filled with `fill`.
249+
fn row(levels: &[u8], axis: RailAxis, fill: u8) -> Vec<u8> {
250+
let mut r = vec![0u8; 512];
251+
for (i, &v) in levels.iter().enumerate().take(RAIL_PAIRS) {
252+
let at = 4 + 2 * i + match axis { RailAxis::Lo => 0, RailAxis::Hi => 1 };
253+
r[at] = v;
254+
}
255+
for b in &mut r[V3_VALUE_OFF..] {
256+
*b = fill;
257+
}
258+
r
259+
}
260+
261+
#[test]
262+
fn depth_stops_at_the_first_hole() {
263+
let spec = RailSpec::v3_facet();
264+
assert_eq!(spec.depth(&row(&[1, 2, 3], RailAxis::Lo, 0)), 3);
265+
// a value AFTER a hole is not ancestry
266+
assert_eq!(spec.depth(&row(&[1, 0, 7], RailAxis::Lo, 0)), 1);
267+
assert_eq!(spec.depth(&row(&[], RailAxis::Lo, 0)), 0);
268+
}
269+
270+
#[test]
271+
fn geodesic_is_the_tree_distance_on_the_linearisation() {
272+
let spec = RailSpec::v3_facet();
273+
let parent = row(&[3, 5], RailAxis::Lo, 0);
274+
let child = row(&[3, 5, 2], RailAxis::Lo, 0);
275+
let sibling = row(&[3, 5, 4], RailAxis::Lo, 0);
276+
let stranger = row(&[9], RailAxis::Lo, 0);
277+
assert_eq!(spec.geodesic(&parent, &child), 1);
278+
assert_eq!(spec.geodesic(&child, &sibling), 2);
279+
assert_eq!(spec.geodesic(&child, &child), 0);
280+
// via the root: depth 3 + depth 1, no shared prefix
281+
assert_eq!(spec.geodesic(&child, &stranger), 4);
282+
// symmetry
283+
assert_eq!(spec.geodesic(&child, &parent), spec.geodesic(&parent, &child));
284+
}
285+
286+
/// The pair is TWO SEPARATE BYTES — a walk on `Lo` must not see `Hi`.
287+
/// This is the test a widened u16 could not pass.
288+
#[test]
289+
fn the_axes_of_a_pair_are_independent() {
290+
let spec_lo = RailSpec::v3_facet();
291+
let spec_hi = RailSpec { axis: RailAxis::Hi, ..spec_lo };
292+
let mut r = vec![0u8; 512];
293+
r[4] = 2; // level 0, Lo
294+
r[5] = 7; // level 0, Hi — different chain entirely
295+
r[6] = 3; // level 1, Lo
296+
assert_eq!(spec_lo.depth(&r), 2);
297+
assert_eq!(spec_hi.depth(&r), 1);
298+
let mut s = r.clone();
299+
s[5] = 9; // change ONLY the Hi axis
300+
assert_eq!(spec_lo.geodesic(&r, &s), 0, "Lo darf Hi nicht sehen");
301+
assert!(spec_hi.geodesic(&r, &s) > 0);
302+
}
303+
304+
/// Stacking a continuation register (e.g. into the edge lane) extends the
305+
/// walk past six levels with the same hole rule.
306+
#[test]
307+
fn a_stacked_register_extends_depth_past_six() {
308+
let plain = RailSpec::v3_facet();
309+
let stacked = plain.stacked(V3_KEY_LEN);
310+
let mut r = vec![0u8; 512];
311+
for i in 0..RAIL_PAIRS {
312+
r[4 + 2 * i] = 1; // primary full
313+
}
314+
r[V3_KEY_LEN] = 1; // continuation level 6
315+
r[V3_KEY_LEN + 2] = 2; // continuation level 7
316+
assert_eq!(plain.depth(&r), 6, "ohne Spec endet die Welt bei 6");
317+
assert_eq!(stacked.depth(&r), 8);
318+
assert_eq!(plain.max_depth(), 6);
319+
assert_eq!(stacked.max_depth(), 12);
320+
}
321+
322+
/// The content distance reads the VALUE and only the value: identical
323+
/// tenants at different addresses are distance 0. The address is where a
324+
/// thing lives, not what it is.
325+
#[test]
326+
fn value_hamming_never_reads_the_address() {
327+
let a = row(&[1, 2, 3], RailAxis::Lo, 0xAA);
328+
let b = row(&[9, 8, 7], RailAxis::Lo, 0xAA);
329+
assert_eq!(v3_value_hamming(&a, &b), 0);
330+
let c = row(&[1, 2, 3], RailAxis::Lo, 0xAB); // 1 bit per value byte
331+
assert_eq!(v3_value_hamming(&a, &c), (512 - V3_VALUE_OFF) as u64);
332+
// truncated row: honest 0, no panic
333+
assert_eq!(v3_value_hamming(&a[..16], &b), 0);
334+
}
335+
336+
/// Triangle inequality spot-check over a small closed set — the property
337+
/// CLAM's pruning actually relies on.
338+
#[test]
339+
fn the_geodesic_satisfies_the_triangle_inequality() {
340+
let spec = RailSpec::v3_facet();
341+
let rows = [
342+
row(&[1], RailAxis::Lo, 0),
343+
row(&[1, 2], RailAxis::Lo, 0),
344+
row(&[1, 2, 3], RailAxis::Lo, 0),
345+
row(&[1, 4], RailAxis::Lo, 0),
346+
row(&[6], RailAxis::Lo, 0),
347+
row(&[], RailAxis::Lo, 0),
348+
];
349+
for a in &rows {
350+
for b in &rows {
351+
for c in &rows {
352+
assert!(
353+
spec.geodesic(a, c) <= spec.geodesic(a, b) + spec.geodesic(b, c),
354+
"Dreiecksungleichung verletzt"
355+
);
356+
}
357+
}
358+
}
359+
}
360+
361+
/// The pseudometric limit, PINNED rather than hidden: distinct nodes
362+
/// deeper than the spec's reach measure 0 apart. Whoever needs identity
363+
/// of indiscernibles must refine over stored ancestry, not trust this.
364+
#[test]
365+
fn distinct_nodes_past_the_stored_depth_measure_zero() {
366+
let spec = RailSpec::v3_facet();
367+
let mut a = vec![0u8; 512];
368+
let mut b = vec![0u8; 512];
369+
for i in 0..RAIL_PAIRS {
370+
a[4 + 2 * i] = 1;
371+
b[4 + 2 * i] = 1;
372+
}
373+
// different VALUES — genuinely distinct nodes, same 6-level prefix
374+
a[V3_VALUE_OFF] = 0xFF;
375+
assert_eq!(spec.geodesic(&a, &b), 0, "das ist die dokumentierte Grenze");
376+
assert!(v3_value_hamming(&a, &b) > 0, "der Inhalt unterscheidet sie");
377+
}
378+
}

‎src/hpc/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ pub mod bnn;
9999
pub mod clam;
100100
#[allow(missing_docs)]
101101
pub mod clam_search;
102+
pub mod clam_v3;
102103
#[allow(missing_docs)]
103104
pub mod clam_compress;
104105
#[allow(missing_docs)]

0 commit comments

Comments
 (0)