From c37137aaacd6bc319b84ae4e1d0c049be945507c Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 18 Oct 2026 11:05:47 +0200 Subject: [PATCH 1/2] docs(research): capture language LEGO grammatical-to-query ABI and current contracts --- .../linguistic-lego-grammar-to-query-rfc.md | 209 ++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 docs/research/linguistic-lego-grammar-to-query-rfc.md diff --git a/docs/research/linguistic-lego-grammar-to-query-rfc.md b/docs/research/linguistic-lego-grammar-to-query-rfc.md new file mode 100644 index 000000000..fd1acb195 --- /dev/null +++ b/docs/research/linguistic-lego-grammar-to-query-rfc.md @@ -0,0 +1,209 @@ +# Language LEGO: corpus-grounded grammar as executable graph-query algebra + +**Status:** RESEARCH / RFC ONLY — not an implemented cross-language query compiler. +**Captured:** 2026-10-18. +**Target:** `lance-graph` SoA ABI, masked RelAlg/Datalog AST, R2IL/Quack, DeepNSM-v2, NAL, and the fixed CausalEdge64 ISA. +**Intent:** Preserve one end-to-end hypothesis and give it a falsifiable landing sequence; do not clone Krill, invent a new parser, redefine CE64 fields, or prematurely add a language runtime. + +## 0. One-sentence thesis + +**Human language is a typed, composable frontend for queries over the same graph on which knowledge and reasoning run.** Interrogatives introduce variables; morphology and syntax constrain their binding; grammatical-function topology constrains permissible semantic interpretations; Datalog/RelAlg executes the resulting graph pattern; NAL/ONA handle supported, uncertain, and competing claims. A Cypher query, a German question, and an English question should lower to the same canonical algebra when they ask the same thing. + +The diagram is not `text → LLM embedding → generated Cypher`. It is: + +```text +COCA / DeReKo [candidate lemma, PoS, frequency, provenance] + UD gold / CorefUD / UniMorph [grammatical and anaphora supervision] + KorAP/Krill/Koral [linguistic spans, positional operators, query AST] + multilingual Bible [verse-aligned *contrasts*, not semantic truth] + │ optional independent input streams + ▼ + Source-specific adapters: token, span, sentence, foundry, annotation, + version, offsets, morphology, candidate alternatives, lineage + ▼ + Canonical grammatical SoA ABI (stable address + columns + masks) + │ + Haspelmath attested function-region masks + dependency + verb-frame + discourse + anaphora constraints + ▼ + Canonical semantic-query AST (variables, roles, scope, negation, + quantifiers, temporal/evidential constraints, projection) + ▼ + RelAlg / Datalog → typed R2IL → mask-risc / Quack / native SoA reads + │ + NAL / ONA oracle-informed, evidence-stamped reasoning + ▼ + Answer rows + provenance + typed abstention / uncertainty + CausalEdge64 only when the shared ABI, witness and evidence gates permit +``` + +The frontend must work for **assertions, questions, explanations, hypotheses, and corrections**, not just for entity mentions. No new natural-language-specific compute universe. + +## 1. Current repository reality and non-duplication + +Read these before implementing anything: + +- `README.md`, "Query paths": Cypher/SQL currently run through DataFusion; Quack's typed builders lower to mask-risc; **there is no production text frontend into Quack**. The experimental Cypher→Quack seam is partial. +- `.arxiv/2026-10_haspelmath-deepnsm-ce64-harness.md` and `crates/deepnsm-v2/src/haspelmath.rs` (#1570): Fig. 8.1 grammatical-function continuity, attested forms, incomplete vs disconnected verdicts, fixed **epoch-wide** operand dictionary and explicit refusal of out-of-ABI cases. Do **not** create a rival Haspelmath implementation. +- `.arxiv/2026-10_haspelmath_semantic_maps_grammar_abi_notes.md` and `docs/research/semantic-morton-qualia-masked-do-rfc.md` (#1545): primary-source limitations; 12×12 / Morton are hypotheses, not results from Haspelmath. +- `crates/lance-graph-quack/examples/semantic_map_probe.rs` and `.claude/board/entries/2026-10-16-grammar-coordinates-annotation-wiring.md` (#1591): plane-based connectedness over additional maps; unification with the `haspelmath.rs` fixtures remains open. +- `crates/deepnsm-v2/src/{coca,lexicon,fsm,tekamolo,loci,wire}.rs` (#1314, #1321, #1591): lexical candidates, multi-reading FSM, German TEKAMOLO hypotheses, local loci, explicit CE64 wiring. No new implementation claimed here. +- `crates/causal-edge/src/orientation.rs` (#1594): bit 43..45 is one global `Orientation` codebook, `Core` plus seven dative functions. Do not invent a private syntax code there. The eighth Haspelmath function, Judicantis, is not a CE64 orientation. +- `.claude/knowledge/epistemic-frontier-capstone.md` and `.claude/board/entries/2026-10-17-ce64-one-layout-moore-synapse-membrane.md` (#1595/#1597): decisions about fixed CE64, layers, reference-frame/spine, presence masks and R2IL. Some decisions supersede still-present **code readers and older docs**; inventory and reconcile in their own PR before production writes. +- `crates/lance-graph-relalg/src/ir.rs`: currently describes string-free selection of one row space using `All/Empty/Plane/Atom/And/Or/Not`; full Datalog joins, recursion and quantified query AST are **not silently present**. +- `crates/lance-graph-nal/src/lib.rs`: typed NAL folds operate over sealed coordinates, evidence stamps and borrowed lanes. ONA/OpenNARS are **comparison oracles**, not an accepted cloned interpreter. + +Existing measurement to retain: `.claude/board/entries/2026-10-04-deepnsm-v2-position-rules-on-ud-gold.md` reports independent UD English/German/French POS probes; `...-german-tekamolo-verb-position.md` reports **TEKAMOLO ordering is not a reliable universal grammar law** on the measured German text. Do not reverse that finding by making Te-Ka-Mo-Lo a mandatory sentence-order parser. + +## 2. Grammatically meaningful LEGO, not token lookup + +A source token is not a concept. Preserve this separation: + +| Level | Example | Representation / invariant | +| --- | --- | --- | +| Surface | German `dem`, English `to` | FormId + source offsets; language-local | +| Lexical | lemma and POS alternatives | multivalued `(LemmaId, UposId, features, prior, source)` | +| Grammar | `Case=Dat`, `obl`, `acl:relcl` | typed UD-compatible dependency facts; no semantic role implied by tag alone | +| Semantic function | RECIPIENT, BENEFICIARY, LOCATION, DIRECTION, PURPOSE | versioned palette/function identity, Haspelmath connectivity constraints | +| Event/role | `GIVE(event)`, `RECIPIENT(event, child)` | independent, canonical relational facts | +| Epistemology | observed, associated, supported, causal candidate, causes | NAL evidence and CE64 read/write authority, not parser plausibility | +| Query | projection `?drug`, negation, every/some, why | executable semantic AST with explicit scope and answer mode | + +German `Dativ` is *not* BENEFICIARY, and German `Akkusativ` is *not* DIRECTION. Cases give constraints that are interpreted under verbs, prepositions, frames and construction context. `auf dem Tisch` vs `auf den Tisch` is a useful minimal adversarial pair. German `zum Bahnhof` is directional despite Dativ. + +Haspelmath (2003, `The Geometry of Grammatical Meaning`, Figs. 8.1 etc.) is about attested **grammatical-function adjacency and multifunctional form regions**, not a 144-verb vocabulary, address system or semantic metric. A complete attested disconnected coverage set can falsify a proposed map; missing/intermittent attestation must stay `Incomplete`. Mere adjacency licenses candidate exploration, not equivalence, entailment or causal certification. + +Proposed codebook rule: **stable semantic identity by versioned globally pinned definition**, never "u8 happens to match". Numeric locality (Morton, 12×12 palette, HHTL) is an execution choice to compare against controls, not proof of semantic proximity. If a function set exceeds an 8-bit local palette, use explicit versioned indirection or a separate wider identity. **Do not widen/relabel CE64 8:8:8 on ingestion.** + +## 3. Preserve existing TEKAMOLO and anaphora ABI + +### TEKAMOLO + +`crates/deepnsm-v2/src/tekamolo.rs` describes an existing **16-byte V3 facet** (`classid 4 + 12-byte value`) read as `G4D3`, four coarse→fine 3-byte lanes **Temporal, Kausal, Modal, Lokal** (4×3×u8). Zero lane = *unaddressed*. Its current German-language heuristic and left-corner/right-corner competition are a producer, not the sole authority. Do not replace its payload by a new `[event,predicate,argument]` struct; that was only an earlier explanatory analogy and would break the actual coarse/fine contract. Add a **semantic event-role projection beside** the existing facet, with explicit mapping. + +The four axes concern modifiers, not the entire predicate-argument graph. Core AGENT, RECIPIENT, THEME are independent relational facts. Kausal/final denotes linguistic reason/purpose; it does **not** certify a Pearl intervention. Temporal in current Bible-oriented hydration may denote verse/TOC position; do not silently promote narrative position to event time. + +### 24×i4 loci + +`lance-graph-contract/src/causal_witness.rs` defines `CausalWitnessFacet`: a **reading** of 12 bytes as 24 signed i4 event-context loci. Zero = unbound (not a zero-distance reference). Sixteen named loci already exist (Temporal, Kausal, Modal, Lokal, S/P/O meaning, Antecedent, BasinAnchor, SupportedBy, Supports, RunbookEvidence, QualiaReference, MeaningLevel, Quorum, Contradiction). Eight are reserved; **do not reassign or turn them into arbitrary token registers**. + +Offsets index the **event/triple stream**, not raw token positions. In `.claude/board/entries/2026-10-16-grammar-coordinates-annotation-wiring.md`, G1 (relative-clause antecedent to locus) and G2 (TEKAMOLO token-to-event binding) are open because the needed mapping is not exported. A local reference outside the signed window is an **external witnessed basin/link**, not a truncated i4. The source/target unit must be proven equal before storing a nibble. + +Test sentence: `Ich gebe dem Kind, das dort wartet, heute gern ein Buch, damit es lesen kann.` Different clause-local case of `das` and its dative antecedent `dem Kind` must not be collapsed; `es` can be grammatically ambiguous between `Kind` and `Buch`. Keep alternatives and selectional evidence. The earlier chat's "24 offsets partitioned evenly across T/K/M/L" is not the in-tree contract. + +## 4. Language-standardized DTO: data contract, not false universal semantics + +**Canonical input DTO** must round-trip source facts plus annotation provenance: + +- corpus ID, version, license/access scope, source hash, foundry/tagger and annotation scheme/version; +- original sentence/verse/document identity, token ordinals, character offsets and segment boundaries; +- language, surface, lemma, UPOS, original XPOS/STTS/CLAWS-like labels, morphological FEATS; +- HEAD/DEPREL, enhanced DEPS, MISC, multiword token ranges and empty nodes; +- candidate analyses, score/count/prior and the exact source reading that supplied each; unresolved/unknown must remain explicit; +- optional CorefUD mentions, entity clusters, discourse connective links, alignment lineage. + +UD **normalizes labels**, not semantic equivalence. CoNLL-U as a serialization does **not** guarantee valid UD: the published `KorAP-XML-CoNLL-U` example writes STTS `VVFIN` into an UPOS-looking column. Validate and explicitly map tags to UD UPOS/FEATS before treating them as gold-compatible. Never overwrite the original foundry label. + +The **SoA hot path** is IDs, presence masks, offset lanes and typed arrays over a sealed canonical address allocation, with strings in a cold lexical/source dictionary. Multiple readings must be rows/masks or candidate groups, not "first row wins" or a silent overwrite. A new corpus calibration requires a new sealed reference version; do not edit the baked spine in place. + +Do not assume all languages align on one tokenization or that one UD dependency tag implies one thematic role. + +## 5. Linguistic operators as the existing algebra + +Extract *semantics*, not engines, from Krill, Koral, UD, Grew and discourse systems: + +| Linguistic operation | Proposed native algebra | Key risk | +| --- | --- | --- | +| `lemma/POS/FEATS` condition | resident selection masks / lane predicates | original foundry mappings, multireadings | +| phrase/proximity | positional interval relation | tokenization and offset concordance | +| contains/overlaps/begins/ends | span interval algebra | exact empty-span/multiword semantics | +| UD dependency | typed relation with source and target ordinal | no foreign-key gather dressed as a mask | +| relative pronoun/coreference | mention/antecedent joins + local locus when addressable | candidate multiplicity, unbounded references | +| temporal, modal, purpose/cause | event-scoped predicate joins | TEKAMOLO role vs causal assertion | +| conjunction/alternative | mask AND/OR on common row space | joins still needed between row spaces | +| not/without | stratified negation / scoped antijoin | distinguish open-world absence from falsehood | +| recursive descendant/reachability | bounded/fixpoint Datalog lowering | termination, provenance and bag/set laws | +| who/which/what | bound variable + projection | answer scope and wh-question typing | +| why/how do we know | evidence-graph traversal and witness projection | explanation is not causal proof | +| might/probably/according to | explicit modality / source constraints | not silently a `CausalEdge64.c` | + +Example (illustrative intended semantic IR, not a current parser): + +```prolog +answer(Drug, Protein, Study) :- + drug(Drug), + inhibits(Drug, Protein), + relation(Protein, implicated_in, bbb_disruption, Claim), + supports(Study, Claim), + population(Study, human), + publication_year(Study, Year), + Year > 2020. +``` + +Natural question: "Which drugs inhibit proteins implicated in blood-brain barrier disruption, supported by human studies after 2020?" The claim binding on `supports` is critical: a study mentioning the protein is not necessarily evidence for the protein→barrier claim. + +**Compilation is not embeddings/autocomplete and not free-form text-to-Cypher.** The syntactic frontend must bind named predicates against an authorized graph schema/codebook, preserve quantified and negated scope, expose competing interpretations, and refuse unbound symbols. Cypher remains an optional alternate frontend, not a required intermediary. + +**Execution invariant:** push down masks and apertures before reading unnecessary columns; use joins and folds where relation content genuinely requires them. The source graph, vocab IDs and results must share stable row/coordinate semantics. A mask is not a graph traversal, and the current `relalg::Rel` implements Boolean selection only; the wider operators require separately gated work. + +## 6. NAL / ONA / CE64: do not turn syntax into truth + +- Grammar candidate scores, lemma frequencies and parser probability are **not** epistemic `F/C`. Source/annotation quality remains in its own lane. +- NAL revisions need evidence stamps and independence. Thousands of translations derived from one base edition are correlated; verse agreement is not a thousand independent truth roots. +- ONA/OpenNARS/Hyperon are **external oracles** for recipe and rule fidelity where already used; no automatic interpreter port. +- CE64's **one layout** is governed by `causal-edge/src/layout.rs` and its corresponding current contract. `Orientation` bits 43..45 already encode seven dative-map function readings plus Core, not the whole grammar. +- Subsequent 2026-10-17 decisions in `epistemic-frontier-capstone.md` interpret 46..49 as activation and 53..58 as thought layer; code and older working-model docs still contain contrary readers (inference mantissa, W witness handle). **Do not claim migration complete or create a new writer until the separately planned contract reconciliation lands.** +- The "same semantic fact" should have **one canonical address**, with source/grammatical evidence and competing epistemic hypotheses as typed lanes at that address; immutable givens must not be rewritten as hypotheses. +- For questions, the system can return `Answer`, `MultipleReadings`, `Abstain`, or `UnsupportedByCurrentGraph`. Do not fabricate a causal explanation when graph evidence only supports association. + +## 7. The experiment that proves or breaks the thesis + +Use an **independently defined semantic-query fixture set** (the corpus itself is not a semantic gold oracle), with German/English/French initially and other languages held out. For each meaning, include: + +1. Paraphrases across languages with distinct case/preposition/word-order constructions. +2. Known near-misses with different quantifier, scope, polarity, tense, role or evidence binding. +3. German Wechselpräposition (`wo` vs `wohin`), dative recipient vs beneficiary, and accusative object vs direction. +4. Relative clauses: antecedent gender/number/case agreement *within the relative clause*, long references outside the i4 window, and ambiguous neuter pronouns. +5. TEKAMOLO event scoping: `damit` as PURPOSE, `weil` as REASON, `so` as MODAL, `da` ambiguous, clause attachment. +6. Haspelmath connected-region held-out languages, disconnected complete counterexamples and incomplete attestations. +7. Natural-language graph questions with `which`, `why`, negation, existential/universal scope, time qualifiers and evidence restrictions. +8. Corpus noise: Bible verse mismatch, missing verse, paraphrastic divergence, source/translation lineage and domain shift to current German/English. + +**Oracle layers must not be mixed:** +- lexical gold: held-out lemma/POS/FEATS accuracy and candidate recall; +- dependency gold: UD/UAS/LAS etc.; +- coreference gold: CorefUD cluster/mention metrics; +- semantic query gold: *separately authored* canonical AST and answer bindings; +- reasoning gold: exact NAL/ONA rule/evidence oracles and independently witnessed claims. + +Baseline arms: (A) dominant corpus PoS; (B) current DeepNSM-v2 FSM; (C) UD facts + typed semantic rules; (D) optional parser/LLM text-to-Cypher baseline; (E) our masked R2IL compiler. Test on *same* inputs/answer sets and report memory, allocations, p50/p95 latency, cold/warm, ambiguity, abstention, AST equivalence, witness completeness and provenance independence. + +Expected deliverable: **for truly equivalent questions, identical canonical query ASTs and answer bindings; for meaningful differences, non-equivalent ASTs.** If a translated sentence is not genuinely equivalent, preserve the difference; never edit gold to force convergence. + +## 8. Proposed gated implementation slices (none claimed shipped by this RFC) + +| Gate | Action | Acceptance / stop | +| --- | --- | --- | +| L-0, documentation | harvest existing in-tree seams and source repos, pin legal restrictions | no duplicated parser/semantic topology; current-code links pinned | +| L-1, SoA DTO | reversible CoNLL-U(+)-to-typed-ID ingest with foundries, masks and multi-reading candidates | round-trip token/annotation metadata; rejected invalid UPOS; no data redistribution | +| L-2, annotation algebra | differential Krill span/proximity/containment and UD dependency predicates | oracle-equivalent sets, offsets, captures; no unjustified mask-only graph hop | +| L-3, local binding | export FSM antecedent event index and token→event association before assigning 24xi4 loci | no wrong-unit pointer, no overflow remap; ambiguity preserved | +| L-4, semantic functions | multilingual Haspelmath coverage fixtures + case/role/TEKAMOLO adapters into immutable codes | explicit incomplete/falsified verdict; preserves canonical epoch dictionary | +| L-5, query AST | grammar questions to typed, scoped Datalog AST; schema binding and refusal | canonical AST equivalence on equivalent cross-language questions, near-miss divergence | +| L-6, lowering | AST → masked RelAlg + R2IL/Quack joins/folds under current substrate rules | results equal a reference relational oracle; no gather/foreign-key shortcut | +| L-7, reasoning | evidential NAL operators over returned graph paths with stamp/gate | no syntax→truth promotion, no false causes, no correlated overcount | +| L-8, performance | benchmark against Krill/Lucene, Tantivy and reference graph queries | no speed/accuracy claim without same fixtures and access patterns | + +**Sequencing:** L-1 and L-2 can be independent. L-3 is already an identified seam in #1591. L-4 can reuse `haspelmath.rs` and `semantic_map_probe.rs`, not fork them. L-5/6 must not pretend `relalg` already handles recursive joins. + +## 9. Non-goals and refusal contracts + +No new CE128; no per-verse operand-byte meanings; no redefinition of 12-byte facets; no indiscriminate re-use of the reserved eight i4 loci; no mandatory reimplementation of Krill/Lucene/UD/ParZu/ANNIS as runtime dependencies; no unsourced reuse of COCA/DeReKo copyrighted corpora; no "Bible as gold semantics"; no commitment to universal 144 predicates before measured validation; no claim all language grammars can be inferred purely from corpus frequency; no ontology identity inferred from spelling, UPOS or Morton proximity. + +## 10. Downstream documents + +Read alongside: +- `docs/research/linguistic-lego-corpus-and-falsification-matrix.md` (external repository harvest, license gates, probe matrix, exact examples). +- `.claude/knowledge/epistemic-frontier-capstone.md` (source-of-truth current decisions). +- `.claude/board/entries/2026-10-16-grammar-coordinates-annotation-wiring.md` (the already-open antecedent/TEKAMOLO event-index gaps). +- `.arxiv/2026-10_haspelmath-deepnsm-ce64-harness.md` (source-faithful topology, explicit ABI refusals). From 96747ac55bfe4b0b5712783544f8068d411fcd93 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 18 Oct 2026 11:08:37 +0200 Subject: [PATCH 2/2] docs(research): harvest linguistic repos and register falsification gates --- ...ic-lego-corpus-and-falsification-matrix.md | 199 ++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 docs/research/linguistic-lego-corpus-and-falsification-matrix.md diff --git a/docs/research/linguistic-lego-corpus-and-falsification-matrix.md b/docs/research/linguistic-lego-corpus-and-falsification-matrix.md new file mode 100644 index 000000000..b13d9ba6c --- /dev/null +++ b/docs/research/linguistic-lego-corpus-and-falsification-matrix.md @@ -0,0 +1,199 @@ +# Language LEGO — repository harvest, legal gates and falsification matrix + +**Status:** source/integration inventory, not a dependency decision or measured result. +**Captured:** 2026-10-18. Companion to [the grammar-to-query RFC](linguistic-lego-grammar-to-query-rfc.md). + +## 1. Distinguish five types of source before selecting any code + +| Kind | What it can establish | What it cannot establish alone | +| --- | --- | --- | +| Corpus frequencies (COCA, DeReKo) | observed distribution of form/lemma/tag by source, domain and method | correct POS in a given sentence, gold dependencies, semantic truth | +| Linguistic treebanks (UD) | *annotated contextual grammar* and gold benchmark targets for supported splits | entity/coreference identity, rhetorical meaning or graph-query truth | +| Parallel translations (multilingual Bible) | comparable verse keys; alternate grammatical realization and apparent semantic agreement | guaranteed sentence-level alignment or independent semantic gold | +| Corpus search/index engines (Krill/Koral/KorAP) | mature positional/span/dependency retrieval operator semantics | semantic reasoner or universal NLP compiler | +| Semantic / discourse inventories (Haspelmath, PDTB, DiMLex, CorefUD) | type-compatible semantic-function hypotheses, discourse/coreference comparison | Pearl causes, full sentence logical forms or universally fixed predicate-address order | + +A billion automatically tagged tokens are still **silver** unless independently validated. Gold UD is contextual grammar, not gold causal inference. A thousand translations sharing a source edition are not a thousand independent observations. + +## 2. Source systems to read, without making them permanent runtimes + +### Lexical priors and grammatical standards + +| Source | Primary URL | Harvest as | Cautions | +| --- | --- | --- | --- | +| COCA, Corpus of Contemporary American English | https://www.english-corpora.org/coca/ | lexical form→lemma/POS alternatives; frequencies, genre-conditioned priors | non-open corpus licensing; CLAWS-like corpus tags are not UPOS; source-specific folds | +| DeReKo, IDS Mannheim | https://www.ids-mannheim.de/digspra/kl/projekte/korpora/ | German lemma/POS morphology, usage frequencies and foundry disagreement | corpus is access controlled; STTS/multiple foundries, not automatically UD gold | +| UD organization | https://github.com/UniversalDependencies | canonical UPOS, FEATS, HEAD, DEPREL/DEPS and multilingual treebanks | annotation labels are normalized, not semantic identities | +| UD German-HDT | https://github.com/UniversalDependencies/UD_German-HDT | large German dependency fixtures | check release, split, per-treebank license | +| UD German-GSD | https://github.com/UniversalDependencies/UD_German-GSD | case/Wechselpräposition and train/dev/test targets | no explicit TEKAMOLO gold lanes | +| UD English-EWT | https://github.com/UniversalDependencies/UD_English-EWT | English dependency/lexeme controls | genre differences from Bible/COCA | +| UD French-GSD | https://github.com/UniversalDependencies/UD_French-GSD | cross-language ambiguity, word order, dative/recipient checks | surface alignments aren't automatic | +| UD tools | https://github.com/UniversalDependencies/tools | validators and UD evaluation oracle | run actual tags through validation, not just CoNLL-U parser | +| CorefUD | https://ufal.mff.cuni.cz/corefud | multilingual mention/cluster annotation and scoring | not interchangeable with dependency treebank schema | +| UniMorph | https://unimorph.github.io/ | cross-language feature inventories | separate namespace and mapping from UD FEATS | +| PARSEME | https://parsemefr.lis-lab.fr/ | multiword expressions and discontinuous lexical units | extra alignment scope | +| Universal PropBank | https://universalpropositions.github.io/ | predicate/argument-role annotation research | inspect exact data before asserting gold coverage | +| UMR | https://umr4nlp.github.io/ | sentence/document meaning representation research | not a drop-in 8:8:8 register | + +### KorAP: copy operator contracts, not its JVM/Perl runtime + +| Repository | URL | Reusable ingredient | Caution | +| --- | --- | --- | --- | +| Kalamar | https://github.com/KorAP/Kalamar | UI inspection, foundry hints (`tt`, `marmot`, `malt`, `spacy`, `ud`), result views | Mojolicious UI, **not** the engine | +| Kustvakt | https://github.com/KorAP/Kustvakt | API access for searches and annotation match metadata | enforces licensing and authorization; do not bypass it | +| Krill | https://github.com/KorAP/Krill | Lucene positional spans, structural/relation searches, match captures | per-position payloads do not map 1:1 to Tantivy | +| Koral | https://github.com/KorAP/Koral | `KoralQuery` JSON-LD AST from corpus QLs (CQP, ANNIS etc.) | query syntax/AST, *not* semantics or NAL | +| KorAP-XML-CoNLL-U | https://github.com/KorAP/KorAP-XML-CoNLL-U | XML ↔ CoNLL-U, foundry/text/offset metadata | experimental converter; published example has **STTS in UPOS-looking column** | +| korapxmltool | https://github.com/KorAP/korapxmltool | modern Java converter/tagger/parser/index exporter; multi-foundry merge, parallel throughput | README declares **GPLv3**, Java 21, experimental combinations | +| Koral-Mapper | https://github.com/KorAP/Koral-Mapper | e.g. `stts-upos.yaml` mappings | context-dependent tag mapping requires extra checks | +| Tantivy | https://github.com/quickwit-oss/tantivy | Rust inverted index, positions, pretokenized streams | positional payload and structural-graph parity need sidecar or new operators | + +**Implementation direction:** use `korapxmltool`'s **Krill/Koral JSON or the original annotated KorAP XML** as interchange material for a prototype; avoid trying to read opaque Lucene segment files in Rust. Tantivy can retrieve lexical candidates, while SoA relational masks and graph edges carry spans, dependencies and witnesses. Differential-test result sets and captures against Krill; no performance claim from term search alone. + +A published `KorAP-XML-CoNLL-U` readme example includes STTS `VVFIN` under the would-be UPOS column. A validator must reject this until explicit normalization maps it to `VERB` with source tag preserved in XPOS/provenance. + +### Grammar, anaphora and discourse prior art + +| Repository / system | URL | What to borrow | +| --- | --- | --- | +| ParZu | https://github.com/rsennrich/ParZu | German dependency grammar, constraint/ranking ideas, Prolog lineage | +| Grew | https://github.com/grew-nlp/grew | linguistic graph-pattern and graph-rewrite test oracle | +| nlprule | https://github.com/bminixhofer/nlprule | Rust rule-based grammatical analyses and disambiguation | +| SMOR | https://www.cis.uni-muenchen.de/~schmid/tools/SMOR/ | German finite-state inflection/compound analysis, **check non-commercial conditions** | +| Stanza | https://github.com/stanfordnlp/stanza | full off-the-shelf UD annotation baseline | +| spaCy | https://github.com/explosion/spaCy | German/English/French parser baseline | +| pocores | https://github.com/discourse-lab/pocores | German rule-based coreference / antecedent candidates | +| discoursegraphs | https://github.com/arne-cl/discoursegraphs | merge syntax, coreference, RST and connective annotations on aligned tokens | +| DiMLex | https://github.com/discourse-lab/dimlex | German discourse connectives, possible PDTB cause/purpose/time/concession senses | +| DiscourseSegmenter | https://github.com/discourse-lab/DiscourseSegmenter | clause/discourse segmentation rules | +| eduseg_de | https://github.com/discourse-lab/eduseg_de | more recent German segmentation comparison | +| evidencegraph | https://github.com/discourse-lab/evidencegraph | graph model of argument/support/attack | +| RSTmulti | https://github.com/discourse-lab/RSTmulti | multi-annotator rhetorical relationship disagreement | + +Discourse connectivity (`weil`, `damit`, `obwohl`, `während`) should become **typed clause-event relations**, not inferred causal certificates. Reference resolution must distinguish a relative pronoun's internal case from the antecedent's main-clause case. Graph-rewrite or Prolog grammars can be **gold/behavioral oracles** even when we never embed their runtimes. + +### Parallel languages and topology + +| Source | URL | Role | +| --- | --- | --- | +| Haspelmath (2003) semantic maps | https://zenodo.org/records/831410 | measured cross-language function *connectivity*; user-provided scanned `SemMaps.pdf` already distilled in two in-repo research notes | +| Multilingual Bible Corpus | https://github.com/christos-c/bible-corpus | verse-aligned language comparison | +| eBible parallel corpus | https://huggingface.co/datasets/DavidCBaines/ebible_corpus | large-scale aligned corpus candidate; inspect license per translation and versification metadata | +| Token-based parallel-Bible typology study (LREC-COLING 2024) | https://aclanthology.org/2024.lrec-main.965/ | published method for multilingual grammatical comparison | +| Penn Discourse Treebank | https://catalog.ldc.upenn.edu/LDC2019T05 | formal temporal, contingency, comparison and expansion sense hierarchy; licensing applies | + +No public-source statistics in this inventory are a newly reproduced experiment. Pin exact corpus releases and checksums when the prototype begins. Every raw text/annotation extract needs rights review; repo source licenses do not transfer permissions to redistributing underlying corpora or trained models. + +## 3. Small, sharp experiments with semantic output fixtures + +### G-1: Morphology is not thematic role + +| Input construction | Gold question/interpretation to test | Wrong shortcut to reject | +| --- | --- | --- | +| `Ich gebe dem Kind ein Buch.` | recipient(child), theme(book) | all datives are LOCATION or BENEFICIARY | +| `Ich helfe dem Kind.` | affected/benefited child, verb-frame-specific | always `iobj → RECIPIENT` | +| `Das Buch liegt auf dem Tisch.` | stative LOCATION(table) | every `auf` means GOAL | +| `Ich lege das Buch auf den Tisch.` | directed GOAL(table) | every `auf` means LOCATION | +| `Ich gehe zum Bahnhof.` | GOAL(station), despite dative | dative blocks direction | +| `Ich kaufe ein Buch für das Kind.` | BENEFICIARY(child) | beneficiary requires dative | + +Rule: UPOS/case/dependency are **admission constraints**, while canonical semantic role requires a verb frame and construction analysis. Add equivalent French/English items and compare canonical roles, not token offsets or word counts. + +### G-2: Anaphora and local signed pointers + +Sentence: `Ich gebe dem Kind, das dort wartet, ein Buch, damit es lesen kann.` + +- `das` in `das dort wartet` is **nominative in the relative clause**, antecedent `dem Kind` is **dative in the main clause**. Different case is valid; do not collapse the two grammatical analyses. +- `es` has more than one grammatically compatible neuter antecedent. Common-sense reading (`Kind` reads) may be preferred by semantic plausibility; retain evidence/alternatives. +- The existing `CausalWitnessFacet::Antecedent` is a signed **event offset**, not a signed token-index difference. Only write when the parse supplies a proven token→event map and the target is addressable within the valid nonzero window. For anything else, produce a witnessed nonlocal reference or abstain. +- Gold CorefUD may provide clusters, but it does not by itself validate which text propositions are true. + +### G-3: Clause-semantic scope, not string matching + +| Phrase | Candidate relation | Guard | +| --- | --- | --- | +| `weil es regnet` | asserted/explained REASON | does not certify interventional cause | +| `damit er lesen kann` | intended PURPOSE | does not imply goal realized | +| `obwohl es regnet` | CONCESSION | does not reverse actual truth | +| `da` | context-dependent causal or spatial/temporal | retain alternatives until clause resolved | +| `so` | manner/degree/discourse depending on context | do not hard-code Modal for every occurrence | +| `morgen` | event TIME | distinguish deictic time from corpus verse address | + +The existing 2026-10-04 German TEKAMOLO probe falsified naïve Te-Ka-Mo-Lo **surface ordering** as a universal deterministic rule. Retain its frequency-vs-position ablations. + +### G-4: Haspelmath topology, not numerical aesthetics + +- Reuse `haspelmath.rs`'s eight named dative functions and `MapVerdict::Incomplete`. +- Reuse the richer `semantic_map_probe.rs` plane fixtures where appropriate (indefinites, reflexive, instrumental/comitative maps); do **not** silently union distinct semantic maps into one dative wheel. +- Measure whether attested multifunctional marker coverage forms a connected subgraph under a **frozen topology**, with complete/incomplete flags. +- Compare proposed 12×12 or Morton embeddings against adjacency-oracle, shuffled labeling, graph embedding and unconstrained controls. Haspelmath does not establish geometric coordinates or quantized distances. +- The CE64 orientation enum contains 0 Core + 7 dative-function readings; **Judicantis is not encodable as a CE64 orientation**. Do not alias it to an unrelated function. + +### G-5: The executable natural-language question + +Use one independently annotated canonical AST and answer-set fixture for each question below, then express it in German, English and French: + +| Question family | AST requirement | Adversarial near miss | +| --- | --- | --- | +| "Which drugs inhibit proteins implicated in BBB disruption?" | project `Drug`, join inhibition and implicated-in via same protein | "Which proteins inhibit drugs?" must reverse direction | +| "...supported by human studies after 2020?" | bind support to the **implicated-in claim edge**, restrict population/time | a study merely mentioning the protein does not qualify | +| "Which proteins are **not** associated with X?" | explicit bounded complement semantics | open-world absence must not become proven `NOT associated` | +| "Did **every** study support X?" | universal quantifier over an explicit study domain | `some` is different | +| "Why is X believed to cause Y?" | explanation witness path and claim status | related/associated is not equivalent to causes | +| "For whom was the book purchased?" | beneficiary argument projection | recipient/beneficiary conflation | +| "Where is the book / where was it moved?" | LOCATION versus GOAL | dative/accusative shortcut | + +Use the following illustrative Datalog oracle for one fixture (not currently supported verbatim by `lance-graph-relalg`): + +```prolog +answer(Drug, Protein, Evidence) :- + drug(Drug), + inhibits(Drug, Protein), + relation(Protein, implicated_in, bbb_disruption, Claim), + supports(Evidence, Claim), + population(Evidence, human), + publication_year(Evidence, Year), + Year > 2020. +``` + +The **goal is AST/answer equivalence for *genuinely equivalent* questions**, AST divergence for non-equivalent questions, and typed abstention for unknown schema symbols, unsupported spans, evidence or remaining ambiguity. A string similarity score does not prove semantic identity. + +## 4. Measurement matrix and kill switches + +| Layer | Baseline/reference | Metric | Immediate stop | +| --- | --- | --- | --- | +| Candidate coverage | source `word_forms.csv` + multi-reading `LexicalEvidence`; UD train | candidate-set recall by lemma/POS, OOV, source coverage | first-row-wins silently discards a possible gold reading | +| POS/morphology | UD held-out treebanks / `ud_pos_eval` | UPOS, lemma, FEATS; calibration by language/genre | same test examples used for threshold tuning and reported test | +| Dependency parsing | Stanza/spaCy + gold UD | UAS, LAS, well-formedness | STTS in UPOS accepted as UD without validation | +| Span / relation retrieval | Krill/Lucene fixtures, KoralQuery operator AST | identical matches, offsets, captures, ordering where defined | Tantivy term hits mistaken for span semantics | +| Coreference | CorefUD + source resolver controls | clusters/mention accuracy, unresolved ratio, long-reference correctness | offset overflow, event/token index mixing | +| Semantic functions | Haspelmath language-held-out map fixtures | connectedness verdict, incomplete vs falsified, exception counts | adjacency promoted to equivalence or truth | +| TEKAMOLO | current cue/FSM baseline and independent semantic label set | role/attachment precision/recall and abstention | surface TEKAMOLO order treated as inviolable | +| Query semantics | manually audited cross-language AST/answer oracles | canonical AST equality, scope and binding correctness | language-specific shortcut changes semantic predicate identity | +| Reasoning | NAL/ONA oracles; provenance-stamped observations | independent-root revision parity, no spurious `Causes` | parser confidence written as `CausalEdge64.c` | +| Execution | current DataFusion Cypher, Krill/Lucene, Tantivy baselines | cold/warm latency, allocations, peak RSS, masks/read bytes, results | speed claims without identical query semantics and corpus slice | + +Execution should honor the substrate's current `no gather/foreign lookup masquerading as a mask` constraint. A dependency graph hop may need a genuinely supported relational operation; don't claim it is free because candidate selection uses bitmaps. + +**Reproducible existing reference commands, not execution claims for this PR:** + +```bash +cargo run --release --manifest-path crates/deepnsm-v2/Cargo.toml --example ud_pos_eval -- TRAIN.conllu TEST.conllu +cargo test --manifest-path crates/deepnsm-v2/Cargo.toml haspelmath +cargo run --release --manifest-path crates/deepnsm-v2/Cargo.toml --example tekamolo_de -- de_gsd-ud-train.conllu de_gsd-ud-test.conllu +``` + +Pin repository revision, corpus/treebank release, source checksums, licenses, resolved wordform candidate counts, train/dev/test separation, execution machine and all command output in the future implementation PR. The documentation PR does not run these commands. + +## 5. Final anti-drift checklist + +- [ ] No "new language runtime" before verifying the existing in-tree lexical, FSM, Haspelmath, loci, TEKAMOLO, R2IL and NAL seams. +- [ ] Corpus/type licenses checked **before data ingestion**, including individual Bible editions. +- [ ] Source annotations preserved alongside normalized UPOS, lemma, morphological and dependency IDs. +- [ ] Stable classid/epoch/codebook boundary; no per-verse reassignment or overflow aliasing. +- [ ] No new CE64 layout, no private field reinterpretations. +- [ ] Relative-pronoun grammatical agreement, pronoun referential identity and factual truth scored independently. +- [ ] Grammar/semantic-map predictions kept distinct from evidence-stamped epistemic inference. +- [ ] Held-out language AND held-out construction controls; negatives/adversarial near misses. +- [ ] AST round-trip/canonical query equivalence tested against a separate semantic gold set. +- [ ] Performance only discussed after exact result equivalence.