From 878455d39cee25830ed675c7f6cee7ed62322f77 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:04:31 -0500 Subject: [PATCH 01/12] docs(specs): fold the diagram-traversal derivation into diagram.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidate the expand/restrict traversal algebra (derived from first principles: the one primitive, edge rule R1, group rule R2, the additive expand and subtractive restrict, renamed-FK relabel fast-path, and the master-part group rule) into diagram.md as a 'Traversal algebra' design rationale — the single source of truth. cascade.md and trace.md now point to it as the downstream and upstream cases (expand down/up), keeping only their operation-specific content. Relocated from datajoint-python's DESIGN-expand-restrict.md (PR #1524), which is closed; the derivation and spec live with the other specs in datajoint-docs. --- src/reference/specs/cascade.md | 2 + src/reference/specs/diagram.md | 104 +++++++++++++++++++++++++++++++++ src/reference/specs/trace.md | 2 + 3 files changed, 108 insertions(+) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 8832567c..de5e9464 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -4,6 +4,8 @@ This document specifies how DataJoint propagates restrictions across the foreign For the user-facing entry points, see [Delete Data](../../how-to/delete-data.md). For dependent concepts, see [Master-Part](master-part.md), [Diagram](diagram.md), and [Data Manipulation](data-manipulation.md). +Cascade is the **downstream** case of the diagram-traversal algebra — the additive traversal `expand(direction="down")`. The rules it uses (the edge rule R1 and the group rule R2), and why they take the form they do, are derived once in the [Diagram spec's Design Rationale](diagram.md#traversal-algebra); this page specifies the cascade-specific behavior on top of them. + ## Overview A *cascade* starts at a (possibly restricted) **seed** table and propagates the restriction to every table that depends on it via foreign keys, so that a delete or preview affects all dependents consistently. Cascade is invoked by: diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 89c94729..81bb7990 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -290,6 +290,110 @@ If a descendant table lives in a schema that hasn't been activated (loaded into --- +## Traversal algebra + +This design rationale derives the traversal operations from first principles. The operational methods above (`cascade`, `trace`, `restrict`) are not three separate features — they are a small, composable algebra over the dependency graph, derived here from first principles so the *rules*, not just the API, are the specification. The unifying model has **one additive traversal** (`expand`, of which `cascade` is the downstream case and `trace` the upstream case) and **one subtractive traversal** (`restrict`), built on **two rules** (an edge rule and a group rule). + +### 1. The one primitive: propagating a restriction across a foreign key + +A **restriction** on a table is a subset of its rows, written as a condition. + +Every foreign key `child -> parent` defines a function: each child row references exactly one parent row. Propagating a restriction across that edge is itself a **restriction** — restrict the neighbor by the restricted table, matched on the foreign-key attributes (`&` with a query expression). It works in either direction: + +- **downstream** (a restriction on the parent, carried to the child): `child & parent_restricted` — the child rows whose parent is in the restricted set. +- **upstream** (a restriction on the child, carried to the parent): `parent & child_restricted` — the parent rows referenced by the restricted child. + +Downstream and upstream are the *same* operation pointed opposite ways along the same foreign key. This is the whole engine; everything below is how the edge rule degenerates, what parts add, and how you accumulate across many edges. + +### 2. The edge rule (R1, referential) + +**Base case — the foreign key is the whole primary key, not renamed, no parts.** The parent's primary key is embedded verbatim in the child's, with the same column names. A primary-key restriction on the parent is a predicate on exactly those columns, which the child also has, by the same names. So: **carry the restriction unchanged** — the identical predicate that selects the parent rows already selects the matching child rows. This is why the same primary-key restriction rides the whole diagram. + +**Complication A — secondary foreign key.** The parent's key lands in the child's *secondary* (non-primary) attributes. A raw predicate still selects the right child rows, but the restriction is no longer a statement about the child's *identity*, so it cannot be promoted to the child's primary key and ridden further. Keep it relational: project the restricted parent to its key and restrict the child by it, matched on the foreign-key columns. + +**Complication B — renamed foreign key.** The referencing columns have different names in the child. The parent's predicate names columns the child lacks. Fix, mechanically: rename the restriction's columns through the foreign key's attribute map before restricting (reverse the rename going upstream). + +> **R1 (edge rule):** propagate a restriction across a foreign-key edge by **restricting** the neighbor by the restricted table (`&`), projected/renamed onto the shared foreign-key columns. When the foreign key is the whole primary key and unrenamed, the projection is the identity and the restriction collapses to "apply the same predicate." + +### 3. The group rule (R2, compositional) + +Part tables add **compositional** integrity on top of referential integrity: a master and its parts are one entity, created and deleted all-or-nothing. + +- **master -> part** needs nothing new — a part carries `-> master` in its primary key, so R1 already sweeps in all parts of a restricted master. +- **part -> master** is the new rule. A restriction landing on part rows satisfies referential integrity by touching just those rows, but leaves a fragment of an entity. So it must **lift existentially to the master** (the master is in if *any* of its parts is), and the master re-expands to **all** its parts. + +> **R2 (group rule):** a restriction touching any part of a master's group brings the whole group — existential lift part -> master, then expand master -> all parts. + +R2 is a closure over the master–part grouping, which is exactly why foreign-key restrictions alone cannot express it. + +### 4. Two operations over R1 + R2 + +There are exactly two irreducible ways to use the rules, and they are opposites. + +**`expand` — additive (grow from one seed).** A constructor: seed a single restricted table and grow outward by R1 + R2, accumulating reachable rows by **union** (a table is reached if reachable via any path). Directional, `direction="down" | "up" | "both"` (default `"down"`): + +- `direction="down"` — descendants: the **delete blast radius**. This is `cascade`. +- `direction="up"` — ancestors: the **valid query sources** a `make()` may read under the reproducibility contract. This is `trace`; inside `make()`, `self.upstream` is `expand(self & key, direction="up")`. +- `direction="both"` — a referentially-consistent **export region** around the seed. + +A single-seed additive closure is always consistent and never needs an intersection: tracing up pulls exactly the referenced ancestors, cascading down pulls exactly the dependents. + +**`restrict` — subtractive (progressively carve any diagram).** An instance method on any diagram, carving it down by applying conditions, accumulating by **intersection** — **every table is restricted by the conjunction of all conditions that reach it**; tables that go empty drop out. It is: + +- **progressive / chainable** — `.restrict(A).restrict(B)…`, each carves further; +- **order-independent** — the result is the conjunction of all conditions; +- **monotone** — every step only removes; every intermediate is a valid slice. + +`restrict` is *not* reducible to combinations of `expand`, because it applies **multiple independent conditions** and gives each table the AND of the ones upstream of it. Example: "all data for `subject_id=5` **and** `method_id=5`" — `Subject` and `ProcessingMethod` are independent ancestors meeting only at a shared descendant; combining single-seed expansions cannot assemble it, chained `restrict` does. + +### 5. Why this is the whole story + +- **Intersection is not a convergence rule.** It only arises in the subtractive model with multiple independent conditions (`restrict`). The additive model (`expand`) is pure reachability — always union. +- **The two compose freely.** A diagram is a set of tables, each holding one row-set. `expand` unions reachable rows in (grow); `restrict` intersects a propagated condition in (carve). One representation, so they chain in any order. A grow step ORs rows in (restrict by a list, `[cond, …]`); a carve step ANDs a further restriction on (chained `&`). +- **Materialization is a delete-time concern, not part of traversal.** Freezing a group's keys before deleting (delete runs parts-before-masters) matters only when a traversal feeds `delete`; the read-only closures never pay for it. + +### 6. Renamed foreign keys and the seed restriction + +Each time a restriction crosses a foreign key it must be re-expressed in the neighbor's attribute names. Renaming is the only thing that changes names across an edge, so it is the only place this needs care, and the *shape* of the seed restriction decides how. A restriction is one of three kinds: + +- **materialized** — a dict of primary-key values, or a sequence of them (literal `attr: value` rows, e.g. `A.keys()`); +- **subquery** — a query expression (another table, possibly restricted); +- **string** — a raw SQL predicate over attribute names, e.g. `'weight > 10'`. + +Only the materialized kind is *frozen literal values*; the other two are *live* (evaluated against current data). This split decides whether a renamed edge is crossed by simply relabelling or must be crossed relationally. + +**Kind 1 — materialized (relabel fast-path).** Crossing a renamed foreign key, the neighbor's restriction is obtained by **renaming the dict's keys through the edge, values unchanged**: keep the referenced attributes (relabelled) and drop any key field that does not exist on the neighbor. Going upstream this drops the child's own identity attributes, leaving exactly the parent's key; going downstream nothing is dropped and the child's own key attributes stay unconstrained (a partial key). Renamings chain, so a key relabels edge by edge. This is exact because the renaming is pure (values and types preserved) and attribute identity across the edge is fixed by the edge's pairing, not by coincidental name matches. A sequence of dicts (e.g. `A.keys()`) is the OR of its members and relabels element by element; the walk stays symbolic — no subqueries — and, being frozen literals, is stable and delete-safe. + +*Which attributes cross an edge:* an attribute propagates across an edge iff that edge **carries** it (the foreign key references it). A data attribute (`weight`) is carried by no edge and never propagates — a key containing one is really an `A & cond` case (Kind 2/3), reducible to Kind 1 by materializing to keys first, `(A & cond).keys()`. A *secondary* foreign-key attribute propagates along its own edge even though it is not part of the primary key. So the test is per edge — *does the key cover the attributes this edge carries?* — not a global "is the key primary-key-only?". + +*Direction, because a foreign key is a function:* **down** (parent to children) is the preimage — a parent key relabels to a partial child key and always suffices. **Up** (child to parent) is the image — to name the referenced parent by relabelling, the key must include the parent's **full primary key** (in the child's names). When it doesn't, `expand` (which must be exact) **materializes** — queries the child for the actual referenced parent keys — while `restrict` (which removes only the provably-excluded) may **leave the parent uncarved** (a looser but still referentially-valid slice). Same relabel condition for both; only the fallback differs. + +**Kinds 2 & 3 — live (restrict-then-project).** A subquery or string restriction has no literal values to relabel; cross the edge **relationally** — restrict `A` by the condition, project it onto the neighbor's referenced attributes (renamed), and restrict the neighbor. A string cannot cross as text (it may name attributes the foreign key doesn't carry, and rewriting SQL to the neighbor's names is not reliable); only its *effect* crosses, through the referenced-attribute values of the surviving `A` rows. Any live restriction becomes relabel-able (and delete-safe) the moment it is materialized to keys, `(A & r).keys()` — which is exactly what `cascade` does at plan time. + +### 7. The group rule and the relabel fast-path + +R2 needs **no new key machinery** — it is the relabel fast-path run twice, with the part-specific attribute deliberately lost in between. For a master `Session` (primary key `session_id`) and part `Session.Trial` (primary key `(session_id, trial_id)`): + +- **master to part (down):** the ordinary relabel — `Session & {'session_id': 5}` becomes the partial key `{'session_id': 5}` on `Session.Trial`, selecting *every* trial of session 5. "All parts follow the master" falls straight out of the down rule. +- **part to master (up), the existential lift:** relabel-drop — `Session.Trial & {'session_id': 5, 'trial_id': 2}` keeps `session_id`, drops the part-specific `trial_id`, giving `{'session_id': 5}`. A sequence of part keys across many trials all drop to the same master key and de-duplicate, so the OR-over-siblings is free. +- **master to all parts (re-expansion):** *not* a relabel of the seed — a fresh downstream step from the recovered master key, `{'session_id': 5}` on `Session` relabels down to `{'session_id': 5}` on `Session.Trial`, which drops the `trial_id` constraint and so *widens* from "trial 2" to all trials. + +**The signature: a key that reaches a part via its master carries no part-specific constraint.** The lift *narrows* the key to the master; the re-expansion *widens* it to all parts; the part-specific attribute is destroyed by the lift and cannot be recovered. That loss *is* compositional atomicity in key terms — the whole part-group comes along precisely because the returning key no longer distinguishes one part from its siblings. + +Corollaries: a materialized-key seed stays materialized through the whole part -> master -> parts round trip, so it is delete-safe for free (the delete-order materialization only ever fires for *live* seeds); and one mechanism serves all three call sites — the mutating part->master cascade, the upstream `trace` that surfaces an ancestor master's parts, and `restrict` promoting a part predicate to its master. + +### Summary + +| | additive (grow) | subtractive (carve) | +|---|---|---| +| operator | `expand(seed, direction)` (`cascade` = down, `trace` = up) | `diagram.restrict(*conditions, direction)` | +| accumulate | union | intersection | +| serves | delete blast radius / `make()` sources / export region | multi-condition pipeline carving | + +One data structure, two composable transforms, two rules — R1 (edge restriction) and R2 (group). `cascade` and `trace` are the downstream and upstream cases of the additive traversal. + +--- + ## Output Methods ### Graphviz Output diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 8a9b78a5..5ca1cd4c 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -12,6 +12,8 @@ The two are designed as a unit. `trace` is the underlying graph operation; `self For the related downstream operation, see [Cascade Specification](cascade.md). For `make()` itself, see [AutoPopulate Specification](autopopulate.md). +Trace is the **upstream** case of the diagram-traversal algebra — the additive traversal `expand(direction="up")`, the mirror of cascade's downstream `expand(direction="down")`. The shared rules (the edge rule R1 and the group rule R2) are derived once in the [Diagram spec's Design Rationale](diagram.md#traversal-algebra); this page specifies the upstream-specific behavior and the `self.upstream` surface on top of them. + ## Why this exists A computed row's reproducibility rests on the convention that `make(self, key)` reads only from its declared upstream dependencies. Making that convention easy to follow — and its result easy to inspect — requires two pieces working together: From 3a824278e4c25745c6b0402e43225a9b456c24c5 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:48:49 -0500 Subject: [PATCH 02/12] docs(specs): collapse the traversal algebra to a single expand operation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop restrict and the additive/subtractive split from the diagram.md derivation. The model is now one operation, expand(seed, direction in {up,down,both}): cascade = down, trace = up, export = both, over the two rules R1 (edge) and R2 (group). A filter over several independent tables is not a second operation — it either seeds a common descendant (which inherits both keys) and expands both ways, or, lacking one, is a union of expansions; the per-table 'AND of upstream conditions' carving is a UI-layer composition of expands, not a core primitive. --- src/reference/specs/diagram.md | 38 ++++++++++++++-------------------- 1 file changed, 15 insertions(+), 23 deletions(-) diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 81bb7990..0f0f6b2a 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -292,7 +292,7 @@ If a descendant table lives in a schema that hasn't been activated (loaded into ## Traversal algebra -This design rationale derives the traversal operations from first principles. The operational methods above (`cascade`, `trace`, `restrict`) are not three separate features — they are a small, composable algebra over the dependency graph, derived here from first principles so the *rules*, not just the API, are the specification. The unifying model has **one additive traversal** (`expand`, of which `cascade` is the downstream case and `trace` the upstream case) and **one subtractive traversal** (`restrict`), built on **two rules** (an edge rule and a group rule). +This design rationale derives the traversal operations from first principles. The operational methods above (`cascade` and `trace`) are not separate features — they are the two directions of **one traversal**, `expand`, over the dependency graph, derived here so the *rules*, not just the API, are the specification. The model is a single operation `expand(seed, direction)` — `cascade` is the downstream case, `trace` the upstream case — built on **two rules**: an edge rule (R1) and a group rule (R2). ### 1. The one primitive: propagating a restriction across a foreign key @@ -326,30 +326,22 @@ Part tables add **compositional** integrity on top of referential integrity: a m R2 is a closure over the master–part grouping, which is exactly why foreign-key restrictions alone cannot express it. -### 4. Two operations over R1 + R2 +### 4. The one operation: expand -There are exactly two irreducible ways to use the rules, and they are opposites. - -**`expand` — additive (grow from one seed).** A constructor: seed a single restricted table and grow outward by R1 + R2, accumulating reachable rows by **union** (a table is reached if reachable via any path). Directional, `direction="down" | "up" | "both"` (default `"down"`): +There is one traversal. Seed a single restricted table and grow outward by R1 + R2, accumulating reachable rows by **union** (a table is reached if reachable via any path). It is directional, `direction="down" | "up" | "both"` (default `"down"`): - `direction="down"` — descendants: the **delete blast radius**. This is `cascade`. - `direction="up"` — ancestors: the **valid query sources** a `make()` may read under the reproducibility contract. This is `trace`; inside `make()`, `self.upstream` is `expand(self & key, direction="up")`. -- `direction="both"` — a referentially-consistent **export region** around the seed. - -A single-seed additive closure is always consistent and never needs an intersection: tracing up pulls exactly the referenced ancestors, cascading down pulls exactly the dependents. - -**`restrict` — subtractive (progressively carve any diagram).** An instance method on any diagram, carving it down by applying conditions, accumulating by **intersection** — **every table is restricted by the conjunction of all conditions that reach it**; tables that go empty drop out. It is: +- `direction="both"` — a referentially-consistent **export region** around the seed: everything the seed rows depend on and everything derived from them. -- **progressive / chainable** — `.restrict(A).restrict(B)…`, each carves further; -- **order-independent** — the result is the conjunction of all conditions; -- **monotone** — every step only removes; every intermediate is a valid slice. +A single-seed closure is always consistent and never needs an intersection: tracing up pulls exactly the referenced ancestors, cascading down pulls exactly the dependents. Every requirement is a case of `expand`: blast radius (`down`), `make()` sources (`up`), and "all data for this entity" / consistent export (`both`). -`restrict` is *not* reducible to combinations of `expand`, because it applies **multiple independent conditions** and gives each table the AND of the ones upstream of it. Example: "all data for `subject_id=5` **and** `method_id=5`" — `Subject` and `ProcessingMethod` are independent ancestors meeting only at a shared descendant; combining single-seed expansions cannot assemble it, chained `restrict` does. +**Multiple conditions do not need a second operation.** A filter over several *independent* tables (e.g. "data for `subject_id=5` **and** `method_id=5`") is still `expand`. Either the conditions share a common descendant that inherits both keys — seed that descendant with the combined condition and `expand(both)` — or they do not, in which case no table is downstream of both, there is nothing to intersect, and the result is the union of the individual expansions. A per-table "conjunction of upstream conditions" carving is a UI-layer composition of `expand`s (a Navigator concern), not a core operation. ### 5. Why this is the whole story -- **Intersection is not a convergence rule.** It only arises in the subtractive model with multiple independent conditions (`restrict`). The additive model (`expand`) is pure reachability — always union. -- **The two compose freely.** A diagram is a set of tables, each holding one row-set. `expand` unions reachable rows in (grow); `restrict` intersects a propagated condition in (carve). One representation, so they chain in any order. A grow step ORs rows in (restrict by a list, `[cond, …]`); a carve step ANDs a further restriction on (chained `&`). +- **It is pure reachability.** A restriction reaches a table if it reaches it via *any* foreign-key path, so convergence is always **union** — there is no intersection to reason about and no second, subtractive operation. +- **One data structure.** A diagram is a set of tables, each holding one row-set; `expand` unions reachable rows into it as it grows. - **Materialization is a delete-time concern, not part of traversal.** Freezing a group's keys before deleting (delete runs parts-before-masters) matters only when a traversal feeds `delete`; the read-only closures never pay for it. ### 6. Renamed foreign keys and the seed restriction @@ -366,7 +358,7 @@ Only the materialized kind is *frozen literal values*; the other two are *live* *Which attributes cross an edge:* an attribute propagates across an edge iff that edge **carries** it (the foreign key references it). A data attribute (`weight`) is carried by no edge and never propagates — a key containing one is really an `A & cond` case (Kind 2/3), reducible to Kind 1 by materializing to keys first, `(A & cond).keys()`. A *secondary* foreign-key attribute propagates along its own edge even though it is not part of the primary key. So the test is per edge — *does the key cover the attributes this edge carries?* — not a global "is the key primary-key-only?". -*Direction, because a foreign key is a function:* **down** (parent to children) is the preimage — a parent key relabels to a partial child key and always suffices. **Up** (child to parent) is the image — to name the referenced parent by relabelling, the key must include the parent's **full primary key** (in the child's names). When it doesn't, `expand` (which must be exact) **materializes** — queries the child for the actual referenced parent keys — while `restrict` (which removes only the provably-excluded) may **leave the parent uncarved** (a looser but still referentially-valid slice). Same relabel condition for both; only the fallback differs. +*Direction, because a foreign key is a function:* **down** (parent to children) is the preimage — a parent key relabels to a partial child key and always suffices. **Up** (child to parent) is the image — to name the referenced parent by relabelling, the key must include the parent's **full primary key** (in the child's names). When it doesn't, the relabel fast-path cannot fire and `expand` **materializes** — queries the child for the actual referenced parent keys — and continues. **Kinds 2 & 3 — live (restrict-then-project).** A subquery or string restriction has no literal values to relabel; cross the edge **relationally** — restrict `A` by the condition, project it onto the neighbor's referenced attributes (renamed), and restrict the neighbor. A string cannot cross as text (it may name attributes the foreign key doesn't carry, and rewriting SQL to the neighbor's names is not reliable); only its *effect* crosses, through the referenced-attribute values of the surviving `A` rows. Any live restriction becomes relabel-able (and delete-safe) the moment it is materialized to keys, `(A & r).keys()` — which is exactly what `cascade` does at plan time. @@ -380,17 +372,17 @@ R2 needs **no new key machinery** — it is the relabel fast-path run twice, wit **The signature: a key that reaches a part via its master carries no part-specific constraint.** The lift *narrows* the key to the master; the re-expansion *widens* it to all parts; the part-specific attribute is destroyed by the lift and cannot be recovered. That loss *is* compositional atomicity in key terms — the whole part-group comes along precisely because the returning key no longer distinguishes one part from its siblings. -Corollaries: a materialized-key seed stays materialized through the whole part -> master -> parts round trip, so it is delete-safe for free (the delete-order materialization only ever fires for *live* seeds); and one mechanism serves all three call sites — the mutating part->master cascade, the upstream `trace` that surfaces an ancestor master's parts, and `restrict` promoting a part predicate to its master. +Corollaries: a materialized-key seed stays materialized through the whole part -> master -> parts round trip, so it is delete-safe for free (the delete-order materialization only ever fires for *live* seeds); and the same mechanism serves both directions — the mutating part->master cascade (`expand` down feeding a delete) and the upstream `trace` (`expand` up) that surfaces an ancestor master's parts. ### Summary -| | additive (grow) | subtractive (carve) | +| direction | operation | serves | |---|---|---| -| operator | `expand(seed, direction)` (`cascade` = down, `trace` = up) | `diagram.restrict(*conditions, direction)` | -| accumulate | union | intersection | -| serves | delete blast radius / `make()` sources / export region | multi-condition pipeline carving | +| `down` | `expand(seed, "down")` = `cascade` | delete blast radius | +| `up` | `expand(seed, "up")` = `trace` | `make()` sources (`self.upstream`) | +| `both` | `expand(seed, "both")` | consistent export region / "all data for this entity" | -One data structure, two composable transforms, two rules — R1 (edge restriction) and R2 (group). `cascade` and `trace` are the downstream and upstream cases of the additive traversal. +One operation, one data structure, two rules — R1 (edge restriction) and R2 (group). `cascade` and `trace` are the downstream and upstream cases of `expand`; a filter over several independent tables composes `expand`s rather than adding a new operation. --- From 9d90553ac2c09d0f0be6d4597f1d72e1364278c0 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:51:14 -0500 Subject: [PATCH 03/12] docs(specs): align cascade/trace with the expand-only derivation cascade.md: DiGraph -> MultiDiGraph; the part-to-master walk enumerates all simple FK paths (not shortest_path) and defers the edge rule to diagram.md's R1. trace.md: recast the convergence table as the two directions of expand (cascade = down, trace = up), drop the restrict/AND row, and defer the rules to diagram.md (R1 + R2). Fixes statements that were stale after the MultiDiGraph migration and the expand-only model. --- src/reference/specs/cascade.md | 8 ++++---- src/reference/specs/trace.md | 19 +++++++++---------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index de5e9464..ed910574 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -17,7 +17,7 @@ Both follow the same propagation rules; only the terminal step (delete vs. count ## Dependency graph -DataJoint loads its FK structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.DiGraph`). The graph encodes two kinds of structure: +DataJoint loads its FK structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.MultiDiGraph` — a multigraph so that parallel foreign keys, including renamed ones, are each their own edge rather than needing synthetic alias nodes). The graph encodes two kinds of structure: | Element | Encodes | |---|---| @@ -89,13 +89,13 @@ The Master is identified by **naming convention** via `dependencies.extract_mast ### Walking the FK path -The walk uses `nx.shortest_path(master, part)` to find the FK chain from Master to Part: +The walk enumerates **every** simple FK path from Master to Part (`nx.all_simple_edge_paths`), so a Part reachable through more than one foreign-key chain — or through parallel edges between the same pair — is restricted through all of them, combined with OR ([#1492](https://github.com/datajoint/datajoint-python/pull/1492)): ``` -Master → [intermediate Part(s)] → Part +Master -> [intermediate Part(s)] -> Part ``` -Each edge along the path fires one upward rule (U1, U2, or U3) per the edge's metadata (`attr_map`, `aliased`). **Intermediate Parts in a Part-of-Part chain are restricted along the way** — not only the Master. +Each edge along a path applies the edge rule R1 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) in the upstream direction, per the edge's metadata (`attr_map`, renamed-or-not). **Intermediate Parts in a Part-of-Part chain are restricted along the way** — not only the Master. ### Materialization at the Master diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 5ca1cd4c..75981bb3 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -27,21 +27,20 @@ Without (1), downstream tools that need row-level lineage (data-lineage viewers, ### Trace as the upstream mirror of cascade -`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"*. The two share the same dependency graph, the same edge model, and most of the same propagation machinery — only the direction differs. +`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"*. They are the two directions of the same traversal (`expand`), over the same dependency graph and the same rules — only the direction differs: -| Method | Direction | Convergence | Question answered | -|---|---|---|---| -| `Diagram.cascade(expr)` | downstream | OR — any FK path taints | What's affected if these rows are deleted? | -| `Diagram.restrict(expr)` | downstream | AND — must satisfy all FK paths | What satisfies all of these conditions? | -| `Diagram.trace(expr)` | **upstream** | **OR** — any FK path contributes | What contributed to these rows? | +| Method | `expand` direction | Question answered | +|---|---|---| +| `Diagram.cascade(expr)` | `down` | What's affected if these rows are deleted? | +| `Diagram.trace(expr)` | `up` | What contributed to these rows? | -`trace` uses OR convergence because an ancestor entity contributes to a child row if it appears via *any* FK path. (An AND-flavored upstream analog — "ancestors that contributed via *every* path" — is not provided in 2.3.) +Convergence is always **union** — an ancestor contributes to a child row if it is reachable via *any* foreign-key path. This is reachability, not an intersection; there is no AND-flavored upstream analog. -### Reusing the propagation primitives +### Reusing the propagation rules -`trace` applies the **upward propagation rules** (`U1`, `U2`, `U3`) defined in the [Cascade Specification](cascade.md#upward-propagation-child-parent), which are the symmetric inverses of `cascade`'s forward rules. Renamed FKs (`.proj()`) are reversed via U2; Part-of-Part chains are walked through naturally; multiple foreign keys between the same pair of tables are each reversed independently. +`trace` applies the **edge rule R1** in the upstream direction — the same rule `cascade` applies downstream — plus the **group rule R2** for master–part. Both are derived once in the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra); a renamed FK is relabelled through the edge's attribute map, Part-of-Part chains are walked naturally, and parallel foreign keys are each handled independently. -This is why `trace` cannot ship before the upward primitives exist in the codebase. As of DataJoint 2.3, the primitives are in place (added with [#1429's cascade fix](https://github.com/datajoint/datajoint-python/pull/1468)), and `trace` is a direct consumer. +This is why `trace` cannot ship before the upstream primitives exist in the codebase. As of DataJoint 2.3 they are in place (added with [#1429's cascade fix](https://github.com/datajoint/datajoint-python/pull/1468)), and `trace` is a direct consumer. ### The `make()` read/write boundary From afa8179d9ea32e509fc002d4d479118a0e21692c Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:51:50 -0500 Subject: [PATCH 04/12] docs(cascade): drop the single-FK-path limitation (fixed by #1492 all-paths walk) --- src/reference/specs/cascade.md | 1 - 1 file changed, 1 deletion(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index ed910574..0f461b0b 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -203,7 +203,6 @@ For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N The following are known, documented behaviors of the cascade engine as shipped: -- **Single FK path (part→master walk).** The upward walk uses `nx.shortest_path` to find the FK chain from a Part to its Master. If a Part reaches its Master through multiple distinct FK chains, restrictions carried by the non-shortest paths are not applied. Workaround: use `part_integrity="ignore"` and perform the additional deletes manually. - **Materialization memory cost.** The master restriction is materialized via `to_arrays()` (required by the reverse-topological delete order — not merely MySQL error 1093; see [Materialization at the Master](#materialization-at-the-master)). The cost is bounded by the number of distinct master rows referenced by the matching parts. Cascade **preview** (`Diagram.cascade(...).counts()`) pays the same materialization cost as an actual delete. - **Empty-match sentinel.** When no master rows match, the master carries an always-false restriction and appears with zero rows in `counts()` and iteration. This is by design, not an error. - **Enforce granularity.** The `part_integrity="enforce"` post-check is **table-level**: it verifies that *some* rows of the master table were deleted whenever part rows were, not that each deleted part row's *specific* master row was deleted. As a result, rare false negatives (an unrelated master row happened to be deleted in the same cascade, masking a genuine orphan) and false positives (deleting already-orphaned part rows whose master was removed earlier) are possible. From 1019d95b64d1f034a99e1d4a300f2de32df11acd Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:59:29 -0500 Subject: [PATCH 05/12] =?UTF-8?q?docs(specs):=20finish=20traversal=20de-du?= =?UTF-8?q?p=20=E2=80=94=20defer=20cascade/trace=20to=20R1/R2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit De-duplicate cascade.md's F1-3/U1-3 per-rule enumerations against the unified edge rule R1 in diagram.md#traversal-algebra, keeping the cascade-specific behavior on top. Rewrite worked examples to name R1's copy/rename/project cases instead of the old rule codes; drop the subset-symbol notation. Reframe part_integrity per the group rule R2: with cascade the master-part group is one item — master-first pulls in all parts, part-first lifts to the master then re-expands to all parts; ignore treats master and part as regular tables. Update trace.md's allowed-table-set contract so an ancestor master's whole part-group is in scope (R2 upstream), replacing the old non-descending note. Sweep residual unicode arrows to ASCII. --- src/reference/specs/cascade.md | 62 +++++++++++++--------------------- src/reference/specs/trace.md | 8 ++--- 2 files changed, 28 insertions(+), 42 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 0f461b0b..842e9a83 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -22,34 +22,20 @@ DataJoint loads its FK structure into a directed acyclic graph (`Connection.depe | Element | Encodes | |---|---| | **Node** | A table, named by its fully-qualified SQL identifier (e.g. ``` `schema`.`table_name` ```). The graph stores the table's primary key set as node data. | -| **Edge `parent → child`** | An FK constraint from `child` to `parent`. Edge data: `attr_map` (dict mapping child's FK columns → parent's referenced columns), `aliased` (true iff any column was renamed), `primary` (true iff the FK is in the child's primary key). | -| **Parallel edges** | A child can reference the same parent through more than one foreign key — for example two renamed (`.proj()`) references — so the same `parent → child` pair may be connected by multiple edges, each carrying its own `attr_map`/`aliased`/`primary`. | +| **Edge `parent -> child`** | An FK constraint from `child` to `parent`. Edge data: `attr_map` (dict mapping child's FK columns -> parent's referenced columns), `aliased` (true iff any column was renamed), `primary` (true iff the FK is in the child's primary key). | +| **Parallel edges** | A child can reference the same parent through more than one foreign key — for example two renamed (`.proj()`) references — so the same `parent -> child` pair may be connected by multiple edges, each carrying its own `attr_map`/`aliased`/`primary`. | The cascade engine operates on a copy of this graph (the `Diagram` class), recording per-table restrictions in `_cascade_restrictions` and the set of restricted attributes in `_restriction_attrs`. -## Restriction propagation rules +## Restriction propagation across an edge -When propagating a restriction across an edge `parent → child`, one of three rules applies. The rule depends on whether the FK renames columns (`aliased`) and on whether the parent's currently-restricted attributes (`parent_attrs`) are contained in the child's primary key (`child_pk`). +Cascade propagates a restriction across each foreign-key edge by the **edge rule R1** — derived in the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra) — restricting the neighbor by the restricted table, projected/renamed onto the shared foreign-key columns. R1 has three degenerate cases, keyed by whether the foreign key renames columns and whether the restricted attributes are the neighbor's whole primary key: -### Forward propagation (parent → child) +- **copy** — non-renamed, and the restricted attributes are the neighbor's primary-key attributes: carry the same predicate unchanged (the columns share names). +- **rename** — renamed foreign key: project the restricted table with its columns renamed through the edge's attribute map to the neighbor's names. +- **project** — non-renamed, but the restricted attributes are *not* (only) the neighbor's primary key: project the restricted table onto the shared foreign-key columns so the join matches on the right columns. -| Rule | Trigger | Effect on child | -|---|---|---| -| **F1. Copy** | `not aliased and parent_attrs` **non-empty** `and parent_attrs ⊆ child_pk` | Child inherits the parent's restriction directly (same attribute names; literal restriction values copy as-is). An empty attribute set takes rule 3. | -| **F2. Aliased rename** | `aliased` | Child's restriction is `parent.proj(**{fk_col: parent_col for fk_col, parent_col in attr_map.items()})` — the parent expression with columns renamed to match the child's column names. | -| **F3. Project** | `not aliased and parent_attrs ⊄ child_pk` | Child's restriction is `parent.proj()` — the parent projected to its primary key. | - -After applying the rule, the child's restricted-attributes set is updated to track what's now constrained on it. The child becomes a propagation source for its own children in the next pass. - -### Upward propagation (child → parent) - -Symmetric inverses of the forward rules. Used by `part_integrity="cascade"` to propagate a Part's restriction up to its Master through the FK chain. Edge metadata is the same; the direction of travel is reversed. - -| Rule | Trigger | Effect on parent | -|---|---|---| -| **U1. Copy** | `not aliased and child_attrs` **non-empty** `and child_attrs ⊆ parent_pk` | Parent inherits the child's restriction directly (shared attribute names). An empty attribute set takes rule 3. | -| **U2. Aliased reverse-rename** | `aliased` | Parent's restriction is `child.proj(**{parent_col: fk_col for fk_col, parent_col in attr_map.items()})` — the child expression with FK columns renamed back to the parent's column names. | -| **U3. Project** | `not aliased and child_attrs ⊄ parent_pk` | Parent's restriction is `child.proj(*attr_map.keys())` — the child projected onto its FK columns (which, when non-aliased, share names with the parent's PK) so the parent restriction joins on the right columns. | +Cascade applies R1 **forward** (parent to child) for the delete/preview walk. After each step the child's restricted-attribute set is updated, and the child becomes a propagation source for its own children in the next pass. For `part_integrity="cascade"`, R1 is also applied **upstream** (child to parent) to lift a Part's restriction to its Master — the same rule with tail and head swapped. ## Cascade flow @@ -74,14 +60,14 @@ Master-Part integrity reflects the contract that a Master row exists for every P | Mode | Behavior | |---|---| | `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. The *intent* is row-level (each deleted Part row should have its Master deleted), but the shipped post-check is **table-level** — see [Limitations](#limitations) for the resulting rare false negatives and false positives. | -| `"ignore"` | No upward propagation; no post-check. Use when the Master row is intentionally preserved and the user has accepted that Part rows may be orphaned. Caller is responsible for the consequences. | -| `"cascade"` | **Upward propagation enabled.** When cascade reaches a Part, the Master is also restricted (via the upward rules below), and the Master then forward-cascades back down to **all** its Parts (siblings of the originating Part included). Used when the user wants the master-part group treated atomically. | +| `"ignore"` | No upward propagation; no post-check. Master and Part are treated as **regular tables** — each is restricted only through its own foreign keys, and a Part row may be left orphaned when its Master row is preserved. Caller is responsible for the consequences. | +| `"cascade"` | **The master-part group is treated as one item.** Whichever member the cascade encounters first, the whole group comes with it: if the **Master** is reached first, all of its Parts are included; if a **Part** is reached first, the restriction is lifted to its Master (via the upward walk below) and then all of the Master's Parts are included (siblings of the originating Part included). This is the group rule R2 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) applied in either direction of travel. | This document focuses on the `"cascade"` mode; the `"enforce"` and `"ignore"` modes do not change the propagation graph. ## Part-to-Master upward propagation -When `part_integrity="cascade"` and the cascade reaches (or starts at) a Part node, the engine triggers an **upward walk** of the FK graph from the Part to its Master, applying the upward rules (U1, U2, U3) at each edge. +When `part_integrity="cascade"` and the cascade reaches (or starts at) a Part node, the engine triggers an **upward walk** of the FK graph from the Part to its Master, applying the edge rule R1 upstream at each edge. Reaching the Master first needs no walk — its Parts are pulled in by ordinary forward propagation. ### Identifying the Master @@ -141,19 +127,19 @@ class Subject(dj.Manual): """ ``` -`Recording`'s columns are `{src_subject, src_session, recording_id}` — none of them are named `subject_id`. The FK from `Subject.Session → Subject.Recording` is aliased. +`Recording`'s columns are `{src_subject, src_session, recording_id}` — none of them are named `subject_id`. The FK from `Subject.Session -> Subject.Recording` is aliased. When `(Subject.Recording & {"recording_id": 5}).delete(part_integrity="cascade")` runs: 1. **Seed-is-Part check.** `extract_master(Recording) == Subject`. Trigger the upward walk. -2. **FK path.** `shortest_path(Subject, Recording) = [Subject, Subject.Session, Subject.Recording]`. -3. **Walk reversed.** - - Edge `Subject.Session → Subject.Recording`: `aliased=True`. Apply **U2** — `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`. - - Edge `Subject → Subject.Session`: `aliased=False`, `child_attrs={subject_id, session_id} ⊆ parent_pk={subject_id}`? No (`session_id` not in parent pk). Apply **U3** — `Subject` is restricted by `Subject.Session.proj(*attr_map.keys())`, projecting the child onto its FK columns; for this primary FK those columns are just `subject_id`, so this is equivalent to `Subject.Session.proj()` projected to `subject_id`. +2. **FK path.** The simple FK path from Master to Part is `[Subject, Subject.Session, Subject.Recording]` (`nx.all_simple_edge_paths`). +3. **Walk reversed** (R1 upstream at each edge). + - Edge `Subject.Session -> Subject.Recording`: renamed FK — the **rename** case. `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`. + - Edge `Subject -> Subject.Session`: not renamed, but the child's restricted attributes `{subject_id, session_id}` are not (only) the parent's primary key `{subject_id}` (`session_id` is not in it) — the **project** case. `Subject` is restricted by `Subject.Session` projected onto its foreign-key columns, which for this primary FK is just `subject_id`. 4. **Materialize Master.** `Subject`'s restriction is fetched into a value tuple; replaces the chained `QueryExpression`. 5. **Forward pass.** Master forward-cascades back down to `Subject.Session` and `Subject.Recording` (and any sibling Parts not on the original path), now with the materialized restriction. -Without the FK walk (the pre-fix behavior), the engine joined `subject_ft.proj() & recording_ft.proj()` on shared attribute names. `Subject` has `subject_id`; `Recording` has `src_subject`. No shared columns → empty restriction → Master not restricted. This is the failure mode from [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1. +Without the FK walk (the pre-fix behavior), the engine joined `subject_ft.proj() & recording_ft.proj()` on shared attribute names. `Subject` has `subject_id`; `Recording` has `src_subject`. No shared columns -> empty restriction -> Master not restricted. This is the failure mode from [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1. ### Example 2: Part-of-Part with no Master reference in PartB @@ -177,15 +163,15 @@ class Master(dj.Manual): """ ``` -`PartB`'s definition references `Master.PartA`, not `master` directly. The FK chain Master → PartA → PartB still exists in the dependency graph (PartA's FK to Master is `aliased=False`; PartA → PartB is also `aliased=False`). +`PartB`'s definition references `Master.PartA`, not `master` directly. The FK chain Master -> PartA -> PartB still exists in the dependency graph (PartA's FK to Master is `aliased=False`; PartA -> PartB is also `aliased=False`). For `(Master.PartB & {"master_id": 1}).delete(part_integrity="cascade")`: -1. Upward walk: PartB → PartA → Master. - - Edge PartA → PartB: `aliased=False`, `child_attrs ⊆ parent_pk`? `child_attrs = {master_id}`, `parent_pk = {master_id, part_a_id}`. Yes ⊆. Apply **U1** — `PartA` inherits `PartB`'s restriction directly. - - Edge Master → PartA: `aliased=False`, `child_attrs = {master_id} ⊆ parent_pk = {master_id}`. Apply **U1** — `Master` inherits `PartA`'s restriction. +1. Upward walk: PartB -> PartA -> Master (R1 upstream at each edge). + - Edge PartA -> PartB: not renamed, and the child's restricted attributes `{master_id}` are within the parent's primary key `{master_id, part_a_id}` — the **copy** case. `PartA` inherits `PartB`'s restriction directly. + - Edge Master -> PartA: not renamed, and `{master_id}` is the parent's primary key `{master_id}` — the **copy** case. `Master` inherits `PartA`'s restriction. 2. Materialize Master. -3. Forward cascade Master → PartA → PartB picks up all sibling rows under `master_id=1`. +3. Forward cascade Master -> PartA -> PartB picks up all sibling rows under `master_id=1`. Without the FK walk (the pre-fix behavior), the engine jumped directly from PartB to Master via `master_ft.proj() & partB_ft.proj()`. PartA was never restricted, and the chain semantics were silently incorrect. This is [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 2. @@ -195,8 +181,8 @@ For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N ## What is not part of this specification -- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and reuses the same upward rules (U1/U2/U3) defined above. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Upstream Trace Specification](trace.md) for `trace`'s API and semantics. -- **Custom propagation rules** (user-defined): not supported. The three forward and three upward rules cover the cases the FK graph can produce. +- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and applies the same edge rule R1 upstream. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Upstream Trace Specification](trace.md) for `trace`'s API and semantics. +- **Custom propagation rules** (user-defined): not supported. R1 and its three cases (copy, rename, project) cover the cases the FK graph can produce. - **Cross-schema cascade**: handled by `dependencies.load_all_downstream()` called from `Diagram.cascade()`; orthogonal to the propagation rules described here. ## Limitations diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 75981bb3..bbfc1389 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -78,7 +78,7 @@ Returns a `Diagram` instance whose nodes are the seed and all of its ancestors ( `trace` mirrors `cascade`: 1. Load the dependency graph via `connection.dependencies.load_all_upstream()` — the upstream analog of `load_all_downstream`, introduced with `Diagram.trace` ([#1423](https://github.com/datajoint/datajoint-python/issues/1423)). It discovers all schemas reachable via reverse FK edges from the seed's schema. -2. Take the seed's restriction and propagate it **upstream** along the FK graph. For each edge `parent → child`, when the child has a restriction, apply the upward rule (`U1`, `U2`, or `U3` per the cascade spec) to derive the parent's restriction. +2. Take the seed's restriction and propagate it **upstream** along the FK graph. For each edge `parent -> child`, when the child has a restriction, apply the edge rule R1 upstream (its **copy**, **rename**, or **project** case — see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) to derive the parent's restriction. 3. Trim the resulting graph to **seed + ancestors only**. Descendants of the seed and unrelated ancestors are not included. 4. Convergence is **OR**: an ancestor entity is included if reachable through *any* FK path from the seed (consistent with how a child row "comes from" any of its FK parents). @@ -176,7 +176,7 @@ trace.counts() # '`imaging`.`__extract_traces`': 1, '`imaging`.`__summary`': 1} ``` -For a renamed-FK case (paralleling [Cascade Spec §Worked Example 1](cascade.md#example-1-part-of-part-with-renamed-fk)), the upward rules reverse the rename so `trace[Ancestor]` returns the ancestor with its native column names regardless of how the seed's columns are named. +For a renamed-FK case (paralleling [Cascade Spec §Worked Example 1](cascade.md#example-1-part-of-part-with-renamed-fk)), R1's **rename** case reverses the rename so `trace[Ancestor]` returns the ancestor with its native column names regardless of how the seed's columns are named. ## 2. `self.upstream` inside `make()` @@ -195,7 +195,7 @@ Once built, the trace diagram is reused for the remainder of the call. The under `self.upstream` exposes: - All declared ancestors of `self` (transitively, including renamed-FK chains). -- The Parts of ancestors that themselves lie on an FK path to `self` — a Part is included only when it is genuinely reachable through the FK graph, not merely because its master is an ancestor. +- **All Parts of every ancestor Master.** By the group rule R2 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)), a master-part group is one item: once an ancestor Master is in the trace, its whole part-group comes with it — not only those Parts that happen to lie on an FK path to `self`. This is the upstream mirror of `part_integrity="cascade"` treating the group atomically. Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. This is the same guarantee `Diagram.trace(...)` provides; `self.upstream` is just the per-`make()` instance of it. @@ -296,7 +296,7 @@ Teams adopt the read surface incrementally: - Source (shipped in 2.3): `src/datajoint/diagram.py` (`Diagram.trace`), `src/datajoint/autopopulate.py` (`AutoPopulate.upstream`). Implemented in [#1471](https://github.com/datajoint/datajoint-python/pull/1471) (trace) and [#1473](https://github.com/datajoint/datajoint-python/pull/1473) (self.upstream), building on the cascade rules from [#1468](https://github.com/datajoint/datajoint-python/pull/1468). - Issues: [#1423](https://github.com/datajoint/datajoint-python/issues/1423) (Diagram.trace), [#1424](https://github.com/datajoint/datajoint-python/issues/1424) (self.upstream). -- [Cascade Specification](cascade.md) — propagation rules (F1/F2/F3 forward, U1/U2/U3 upward) shared with `trace`. +- [Cascade Specification](cascade.md) — the downstream case of the same edge rule R1 and group rule R2 shared with `trace`. - [AutoPopulate Specification](autopopulate.md) — `make()` execution model and the make() reproducibility contract. - [Diagram Specification](diagram.md) — graph operations on the dependency graph. - [Entity Integrity](../../explanation/entity-integrity.md) — schema dimensions and FK semantics. From e54a70d58f06a9d08c8ebd72237fadba1a24ae12 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 18:00:31 -0500 Subject: [PATCH 06/12] docs(cascade): document delete-time materialization as a general rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Downward expansion for delete cannot leave a restriction referencing a downstream table: reverse-topological delete empties it first, so the subquery matches zero rows and strands what should be deleted (#1496). State this as a general Delete-time materialization section — seed-referencing-descendant and the part-to-master master-key freeze are its instances — applied on every backend by delete-order reason, not as a MySQL-only (error 1093) concern. Preview/counts pays none of it. --- src/reference/specs/cascade.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 842e9a83..b8f83740 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -53,6 +53,16 @@ flowchart TB The engine performs **multiple passes** when `part_integrity="cascade"` is in effect: each pass forward-propagates over all `allowed_nodes`, and a pass may pull in a new master (with its descendants) that requires another pass. The loop terminates because the graph is a DAG and only finitely many nodes can be added. +## Delete-time materialization + +Downward expansion for **delete** carries a constraint that preview (count) does not: a table's restriction must not reference a table that is deleted *earlier* in the plan. + +`Table.delete` executes its per-table deletes in **reverse-topological order** (leaves first — see [Data Manipulation](data-manipulation.md)), so every downstream table is emptied before the table it depends on. If a table's restriction were left as a `QueryExpression` referencing a downstream table — which happens whenever the cascade **seed** is itself restricted by a descendant ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)), or when the master's restriction is left pointing at an already-walked part — that subquery would run against a table already emptied by an earlier step, match zero rows, and silently strand the rows that should have been deleted. + +The fix is to **materialize** any such restriction to a literal key set at plan time, before any row is deleted, while the referenced table still holds its rows. Materialization is applied by delete-order reason on **every backend**, not made backend-conditional: the reverse-topological ordering strands rows regardless of engine. (MySQL additionally rejects a DELETE whose subquery targets the table being modified — error 1093 — but PostgreSQL, which permits that self-reference, still needs materialization for the ordering reason, so this is not a MySQL-only concern.) Where materialization is refused, the engine fails closed with a legible error rather than deleting through a stale subquery. Preview/`counts()` issues no deletes and so pays none of this cost. + +The master-key materialization in [Part-to-Master upward propagation](#materialization-at-the-master) below is the specific instance of this rule for the `part_integrity="cascade"` walk. + ## `part_integrity` modes Master-Part integrity reflects the contract that a Master row exists for every Part row that references it. Three modes govern how cascade enforces or relaxes that contract: @@ -87,9 +97,7 @@ Each edge along a path applies the edge rule R1 (see the [Diagram spec's Travers After the upward walk completes, the Master's accumulated restrictions are **materialized** to a literal value tuple via `(master_ft & restrictions).proj().to_arrays()` and stored as a single condition. Subsequent forward propagation from the Master back down to its other Parts then generates `WHERE pk IN (literal-list)` rather than `WHERE pk IN (SELECT ... FROM )`. -**Why materialization matters.** This is required for correctness on **every backend**, not merely to satisfy MySQL. `Table.delete` executes per-table deletes in reverse-topological order (leaves first — see [Data Manipulation](data-manipulation.md)), so the originating Part is deleted *before* the Master. If the Master's restriction were left as a `QueryExpression` referencing that Part, the Master's own DELETE — issued last — would find the Part already emptied, match zero rows, and silently strand the Master (the very compositional-integrity violation the upward walk exists to prevent). Materializing the Master's primary keys to a literal value set at plan time, before any rows are deleted, captures them while the Part still exists. - -A secondary consequence: the literal set also avoids a self-referential subquery. Left as a query, the Master forward-cascading back to the originating Part would generate a DELETE whose subquery targets the table being modified — which MySQL rejects ("error 1093: You can't specify target table 'T' for update in FROM clause"). PostgreSQL permits that self-reference, but the reverse-topological ordering above means materialization is required on **both** backends regardless — so this must not be treated as a MySQL-only concern. +This is the [delete-time materialization](#delete-time-materialization) rule applied here: the originating Part is deleted before the Master (reverse-topological order), so a Master restriction left pointing at that Part would match zero rows and strand the Master — the very compositional-integrity violation the upward walk exists to prevent. Capturing the Master's primary keys while the Part still holds its rows avoids that, and as a bonus avoids the self-referential subquery MySQL rejects with error 1093. Intermediate Parts in the chain are **not** materialized — they appear only as restrictions on the path, not as forward-cascade sources, so the self-reference issue doesn't arise there. From 7c23bfd14aca30fd12653d6f9e5e51b01a925bc8 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 16:50:10 -0500 Subject: [PATCH 07/12] docs(specs): define Diagram.expand normatively and add the shape catalog Replace the draft "Traversal algebra" section of diagram.md with the normative traversal specification: base principles P1-P4, the row model (foreign-key references, NULLs, parallel foreign keys, the owning reference A1), the operations D, U and U(D(S)) with the delete modes, trace and self.upstream, derived results T1-T8 with justifications, the computation (edge rule, group rule, one pass over the group quotient, groups that iterate, seeds, the plan, delete-time materialization, concurrency), the contract for expanded diagrams, and the retirement of restrict with migration recipes. Remove the restrict/AND-convergence text and the Rule 1-3 propagation section from Operational Methods, which now documents expand, cascade, trace, indexing, counts, prune, iteration and the deprecated restrict. Add traversal-shapes.md: one worked example per schema shape (S1-S7, G1-G6, K1-K6, X1-X2) with the exact rows expand returns. --- mkdocs.yaml | 1 + src/reference/specs/diagram.md | 422 +++++++++---- src/reference/specs/index.md | 1 + src/reference/specs/traversal-shapes.md | 804 ++++++++++++++++++++++++ 4 files changed, 1089 insertions(+), 139 deletions(-) create mode 100644 src/reference/specs/traversal-shapes.md diff --git a/mkdocs.yaml b/mkdocs.yaml index 29b5c7d8..624f9925 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -119,6 +119,7 @@ nav: - Primary Keys: reference/specs/primary-keys.md - Fetch API: reference/specs/fetch-api.md - Diagram: reference/specs/diagram.md + - Traversal Shapes: reference/specs/traversal-shapes.md - Type System: - Types: reference/specs/type-system.md - Codec API: reference/specs/codec-api.md diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 0f0f6b2a..41b27f3b 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -1,10 +1,10 @@ # Diagram Specification -Schema visualization as directed acyclic graphs. +Schema visualization as directed acyclic graphs, and traversal of the rows they connect. ## Overview -`dj.Diagram` visualizes DataJoint schemas as directed graphs showing tables and their foreign key relationships. It provides multiple output formats including SVG, PNG, and Mermaid syntax. +`dj.Diagram` visualizes DataJoint schemas as directed graphs showing tables and their foreign key relationships. It provides multiple output formats including SVG, PNG, and Mermaid syntax. It also computes, for a restricted table, the rows that depend on it or that it depends on: the rows a delete must remove, the inputs of a `make()`, or a self-contained export. ## Design Principles @@ -12,6 +12,7 @@ Schema visualization as directed acyclic graphs. 2. **Graph algebra**: Combine and filter diagrams with set operators 3. **Visual encoding**: Table tiers distinguished by shape and color 4. **Flexible layout**: Configurable direction and schema grouping +5. **One traversal**: `cascade` and `trace` are the two directions of `expand`, defined once in the [Traversal specification](#traversal-specification) --- @@ -92,6 +93,9 @@ dj.Diagram(ComputedTable) - 10 **Note:** Order matters. `diagram + 1 - 1` may differ from `diagram - 1 + 1`. +!!! version-added "New in 2.4" + On a diagram returned by `expand()`, `cascade()`, or `trace()`, only `d1 + d2` of two diagrams with the same direction is defined; the other operators raise. See [Expanded diagrams](#expanded-diagrams). + ### Collapsing Schemas !!! version-added "New in 2.1" @@ -122,106 +126,109 @@ dj.Diagram(Subject) + dj.Diagram(analysis).collapse() !!! version-added "New in 2.2" Operational methods (`Diagram.cascade()`, `restrict`, `counts`, `prune`) were added in DataJoint 2.2. -Diagrams can propagate restrictions through the dependency graph and inspect affected data using the graph structure. These methods turn Diagram from a visualization tool into a graph computation and inspection component. All mutation operations (delete, drop) are executed by `Table.delete()` and `Table.drop()`, which use Diagram internally. +!!! version-added "New in 2.4" + `Diagram.expand()` is the one traversal behind `cascade()` and `trace()`. Its rows are defined in the [Traversal specification](#traversal-specification) below. -### `Diagram.cascade()` (class method) +A diagram can compute, for a restricted table, the rows a delete must remove, the rows a `make()` may read, or the rows an export must carry. The methods in this section compute and inspect those rows without changing any data. `Table.delete()` and `Table.drop()` use them internally and carry out the mutation. + +### `Diagram.expand()` (class method) ```python -dj.Diagram.cascade(table_expr, part_integrity="enforce") +dj.Diagram.expand(seed, direction="down") ``` -Create a cascade diagram for delete. Builds a complete dependency graph from the table expression, includes all descendants across all loaded schemas, propagates the restriction downstream using **OR** semantics — a descendant row is marked for deletion if *any* ancestor path reaches it — and **trims** to the cascade subgraph. - | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `table_expr` | QueryExpression | — | A restricted table expression (e.g., `Session & 'subject_id=1'`) | -| `part_integrity` | str | `"enforce"` | Master-part integrity policy | - -**Returns:** New `Diagram` containing only the seed table and its descendants, with cascade restrictions applied. +| `seed` | QueryExpression | — | One table with any restriction (see [Seeds](#seeds)) | +| `direction` | str | `"down"` | `"down"`, `"up"`, or `"both"` | -**`part_integrity` values:** +**Returns:** a `Diagram` whose nodes are the tables the expansion reaches, each holding the rows of that table in the expansion. -| Value | Behavior | -|-------|----------| -| `"enforce"` | Default. The preview itself never errors; master-part integrity is enforced by `Table.delete()`'s post-check, which rolls back the delete if part rows would be removed without their masters (see [Cascade Spec](cascade.md#part_integrity-modes)). | -| `"ignore"` | Allow deleting parts without masters | -| `"cascade"` | Propagate restriction upward from part to master, then re-propagate downstream to all sibling parts | +| `direction` | Rows | Used for | +|---|---|---| +| `"down"` | `D(S)`: the seed rows and every row that depends on them | delete, impact analysis | +| `"up"` | `U(S)`: the seed rows and every row they depend on | the inputs of `make()`, lineage | +| `"both"` | `U(D(S))`: a region around the seed that is consistent on its own | export | -With `"cascade"`, the restriction flows **upward** from a part table to its master: the restricted part rows identify which master rows are affected, those masters receive a restriction, and that restriction propagates back downstream through the normal cascade — deleting the entire compositional unit (master + all parts), not just the originally matched part rows. +`S` is the set of rows the seed selects. `D`, `U`, and `U(D(S))` are defined under [Operations](#operations). ```python -# Preview cascade impact across all loaded schemas -dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts() -``` +# Every row that depends on subject 1, by table +dj.Diagram.expand(Subject & {"subject_id": 1}, "down").counts() -`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. +# Everything a selected session was derived from +dj.Diagram.expand(Session & {"subject_id": 1, "session_id": 2}, "up")[Subject] +``` -### `Diagram.trace()` (class method) +### `Diagram.cascade()` (class method) ```python -dj.Diagram.trace(table_expr) +dj.Diagram.cascade(table_expr, part_integrity="enforce") ``` -The **upstream mirror of `cascade()`**. Where `cascade()` walks downstream to every descendant, `trace()` walks **upstream** from a (possibly restricted) table expression to every ancestor across all loaded schemas, propagating the restriction along the way. Like `cascade()`, convergence is **OR** — an ancestor is included if reachable through *any* FK path from the seed. +The preview of `(table_expr).delete(part_integrity=...)`. It loads every schema downstream of the seed and holds the rows the delete would remove. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `table_expr` | QueryExpression | — | A restricted table expression whose ancestry is traced | +| `table_expr` | QueryExpression | — | The seed, e.g. `Session & {"subject_id": 1}` | +| `part_integrity` | str | `"enforce"` | Master-part integrity policy | + +| `part_integrity` | Rows | +|---|---| +| `"cascade"` | `D(S)`, the same rows as `expand(table_expr, "down")` | +| `"ignore"` | `D1(S)`: the rows reached through foreign keys alone, with part tables treated as plain tables | +| `"enforce"` (default) | `D1(S)`. `Table.delete()` deletes these rows only if they include the master of every part row among them, and otherwise raises before deleting anything | -**Returns:** New `Diagram` containing the seed and its ancestors, each pre-restricted through the FK join path. +`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. The [Cascade Specification](cascade.md) covers the delete itself. -Access a pre-restricted ancestor by indexing the trace: +!!! version-changed "Changed in 2.4" + `"enforce"` checks each part row's own master, before any row is deleted. In 2.3 it was a table-level check run after the delete. With `"cascade"`, a master joins the delete only if one of its part rows did ([#1591](https://github.com/datajoint/datajoint-python/issues/1591)). ```python -trace = dj.Diagram.trace(MyChild & key) -session = trace[Session] # Session restricted to ancestors of MyChild & key -session.fetch1('session_date') +# Preview the impact of a delete across all loaded schemas +dj.Diagram.cascade(Session & {"subject_id": 1}).counts() ``` -`trace()` reuses the same **upward propagation rules** (U1/U2/U3) documented in the [Cascade Spec](cascade.md#upward-propagation-child-parent). The [Upstream Trace Specification §1](trace.md#1-diagramtracetable_expr) is the normative spec for its API and semantics; inside `make()`, `self.upstream` is the per-`key` instance of a trace. - -### `restrict()` +### `Diagram.trace()` (class method) ```python -diag.restrict(table_expr) +dj.Diagram.trace(table_expr) ``` -Select a subset of data for export or inspection. Starting from a restricted table expression, propagate the restriction downstream through all descendants using **AND** semantics — a descendant row is included only if *all* restricted ancestors match. The full diagram is preserved (ancestors, unrelated tables) so that `restrict()` can be called again from a different seed table, building up a multi-condition subset incrementally. - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `table_expr` | QueryExpression | — | A restricted table expression | +The rows `table_expr` depends on: `U(S)`, the same rows as `expand(table_expr, "up")`. It loads every schema upstream of the seed. The [Upstream Trace Specification](trace.md) covers its use and `self.upstream`, the per-`make()` instance of a trace. -**Returns:** New `Diagram` with restrict conditions applied. The graph is not trimmed. +```python +trace = dj.Diagram.trace(MyChild & key) +trace[Session].fetch1("session_date") +``` -**Constraints:** +!!! version-changed "Changed in 2.4" + A trace includes all parts of every master it reaches, and the ancestors of those parts ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). In 2.3 it included a part only if the part lay on a foreign-key path to the seed. -- **Chainable** — call multiple times to add conditions from different seed tables -- Cannot be called on a Diagram produced by `Diagram.cascade()` -- `table_expr.full_table_name` must be a node in the diagram +### Indexing: `d[T]` ```python -# Chain multiple restrictions (AND semantics) -diag = dj.Diagram(schema) -restricted = (diag - .restrict(Subject & {'species': 'mouse'}) - .restrict(Session & 'session_date > "2024-01-01"')) +d[Session] # a Table class or instance +d["Session"] # a class name, or a full table name ``` +Returns table `T` restricted to its rows in the expansion, as a query expression, in every direction. It raises `DataJointError` if `T` is not a node of the diagram. A node can hold zero rows; indexing it returns an empty expression. + +!!! version-changed "Changed in 2.4" + In 2.3, indexing worked only on diagrams returned by `trace()`. + ### `counts()` ```python diag.counts() ``` -Return affected row counts per table without modifying data. Works with both `cascade()` and `restrict()` restrictions. +Returns the number of rows each node holds, without modifying data. -**Returns:** `dict[str, int]` — mapping of full table names to affected row counts. - -**Requires:** `Diagram.cascade()` or `restrict()` must be called first. +**Returns:** `dict[str, int]` — full table name to row count. Tables in the diagram with no rows map to 0. ```python -counts = dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts() +counts = dj.Diagram.cascade(Session & {"subject_id": 1}).counts() # {'`lab`.`session`': 3, '`lab`.`trial`': 45, '`lab`.`processed_data`': 45} ``` @@ -231,23 +238,15 @@ counts = dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts() diag.prune() ``` -Remove tables with zero matching rows from the diagram view. This only affects the diagram object — no tables or data are modified in the database. Without prior restrictions, removes physically empty tables from the diagram. After `restrict()`, removes tables where the restricted query yields zero rows. - -**Returns:** New `Diagram` with empty tables removed. - -**Constraints:** Cannot be used on a Diagram produced by `Diagram.cascade()`. Cascade diagrams must retain all descendant tables because a table empty at cascade time could have rows by the time `delete()` executes. +Removes the nodes that hold zero rows from the diagram. No table or data is modified. On a diagram that was not expanded, it removes physically empty tables. -**Note:** Queries the database to determine row counts. The underlying graph structure is preserved — subsequent `restrict()` calls can still seed at any table in the schema. +**Constraints:** `prune()` raises on a `down` diagram, including any diagram returned by `cascade()`. A delete must keep every table it reaches, because a table that is empty when the diagram is built can gain rows before the delete runs. `prune()` is allowed on `up` and `both` diagrams. ```python -# Export workflow: restrict, prune, visualize -export = (dj.Diagram(schema) - .restrict(Subject & {'species': 'mouse'}) - .restrict(Session & 'session_date > "2024-01-01"') - .prune()) - -export.counts() # only tables with matching rows -export # visualize the export subgraph +# Export region around one subject, without empty tables +region = dj.Diagram.expand(Subject & {"subject_id": 1}, "both").prune() +region.counts() +region # display the region ``` ### Iteration @@ -259,130 +258,271 @@ Diagrams support iteration in topological order: | `for ft in diagram` | Parents first | Data export, inspection | | `for ft in reversed(diagram)` | Leaves first | Cascade delete, drop | -Each iteration yields a `FreeTable` with any cascade or restrict conditions applied. Only nodes in the diagram's visible set (`nodes_to_show`) are yielded. +Each iteration yields a `FreeTable` restricted to the node's rows. Only nodes in the diagram's visible set (`nodes_to_show`) are yielded. `Table.delete()` and `Table.drop()` use `reversed(diagram)` internally to execute mutations in safe dependency order. -### Restriction Propagation +### `restrict()` + +!!! version-deprecated "Deprecated in 2.4, removed in 3.0" + `Diagram.restrict()` emits a `DeprecationWarning`. Its behavior is frozen. Use `Diagram.expand()`; see [Retiring `restrict`](#retiring-restrict) for the migration. -When `cascade()` or `restrict()` propagates a restriction from a parent table to a child table, one of three rules applies depending on the foreign key relationship: +```python +diag.restrict(table_expr) +``` -**Rule 1 — Direct copy:** When the foreign key is non-renamed and the restriction attributes are a subset of the child's primary key, the restriction is copied directly to the child. +Restricts a full diagram for export: the restriction propagates downstream, and where several restricted ancestors meet, a row is kept only if it matches all of them (AND). The full diagram is preserved, including ancestors and unrelated tables, so `restrict()` can be chained from other seeds. It cannot be called on a diagram produced by `cascade()`. -**Rule 2 — Renamed projection:** When the foreign key uses attribute renaming (e.g., `subject_id` → `animal_id`), the parent is projected with the attribute mapping to match the child's column names. +### Unloaded schemas -**Rule 3 — Full projection:** When the foreign key is non-renamed but the restriction uses attributes not in the child's primary key, the parent is projected (all attributes) and used as a restriction on the child. +`expand()`, `cascade()`, and `trace()` first load every schema reachable through foreign keys in the direction of travel, including schemas the current process has not activated ([X1](traversal-shapes.md#x1-unloaded-downstream-schema)). If a delete still fails with a foreign-key error, DataJoint names the two likely causes: a downstream schema the connection cannot see, or a row inserted concurrently. -**Convergence behavior:** +--- -When a child table has multiple restricted ancestors, the convergence rule depends on the mode: +## Traversal specification -- **`cascade()` (OR):** A child row is affected if *any* path from a restricted ancestor reaches it. This is appropriate for delete — if any reason exists to delete a row, it should be deleted. -- **`restrict()` (AND):** A child row is included only if *all* restricted ancestors match. This is appropriate for export — only rows satisfying every condition are selected. +!!! version-added "New in 2.4" + This section is the normative definition of `Diagram.expand()`, `cascade()`, and `trace()`. The test suite checks the engine against an executable copy of these definitions, and the [Traversal Shape Catalog](traversal-shapes.md) works through one example per schema shape. -**Multiple foreign keys to the same parent:** +The definitions are about rows; the computation is about tables and restrictions. -When a child table references the same parent through multiple foreign keys (e.g., `source_mouse` and `target_mouse` both referencing `Mouse`), these paths always combine with **OR** regardless of the propagation mode. Each foreign key path is an independent reason for the child row to be affected — this is structural, not operation-dependent. +### Base principles -**Unloaded schemas:** +- **P1, referential integrity.** Every foreign-key value names an existing parent row. +- **P2, compositional integrity.** A master row and its part rows are one entity. They are inserted together and deleted together. +- **P3, reproducibility.** `make(key)` reads only rows that `key` depends on. +- **P4, minimality.** An operation touches no row it does not have to. -If a descendant table lives in a schema that hasn't been activated (loaded into the dependency graph), the graph-driven delete won't know about it. The final `DELETE` on the parent will fail with a foreign key error. DataJoint catches this and produces an actionable error message identifying which schema needs to be activated. +### Model and assumptions ---- +**Foreign-key references.** A foreign key `e` of a child table `C` references a parent table `P`. Its attribute map `attr_map_e` pairs attributes of `C` with the primary-key attributes of `P`; the names differ when the declaration renames them with `.proj()`. For a row `c` of `C`, `ref_e(c)` is the row of `P` whose primary key equals `c`'s values on those attributes. By P1, that row exists. + +**A2, NULLs.** If any of `c`'s attributes in `e` is NULL, `ref_e(c)` is undefined: `c` references nothing through `e`. This is SQL's MATCH SIMPLE rule, which both supported backends apply: a foreign key with a NULL column is not checked. Foreign keys can share columns ([S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column)), so a row can have some columns of a foreign key set and others NULL. Such a row still references nothing through that foreign key. -## Traversal algebra +**Parallel foreign keys.** Two foreign keys between the same pair of tables are separate references, each with its own `attr_map` ([S5](traversal-shapes.md#s5-parallel-foreign-keys)). -This design rationale derives the traversal operations from first principles. The operational methods above (`cascade` and `trace`) are not separate features — they are the two directions of **one traversal**, `expand`, over the dependency graph, derived here so the *rules*, not just the API, are the specification. The model is a single operation `expand(seed, direction)` — `cascade` is the downstream case, `trace` the upstream case — built on **two rules**: an edge rule (R1) and a group rule (R2). +**A1, the owning reference.** A part's heading contains its master's primary key under the master's attribute names, with no NULLs, inherited through `-> master` or through a foreign key to a sibling part that satisfies A1 itself. For a part row `r`: -### 1. The one primitive: propagating a restriction across a foreign key +- `master(r)` is the master row with that key; +- `inst(r)`, the **instance** of `r`, is `master(r)` together with every part row, in any part of the master, that has the same master key. -A **restriction** on a table is a subset of its rows, written as a condition. +For a master row `m`, `inst(m)` is `m` with all its part rows. For a row `r` of a plain table, `inst(r) = {r}`. -Every foreign key `child -> parent` defines a function: each child row references exactly one parent row. Propagating a restriction across that edge is itself a **restriction** — restrict the neighbor by the restricted table, matched on the foreign-key attributes (`&` with a query expression). It works in either direction: +A part that violates A1 gets a warning at declaration and is traversed as a plain table: it has no master and no instance ([G6](traversal-shapes.md#g6-a-part-violating-a1)). -- **downstream** (a restriction on the parent, carried to the child): `child & parent_restricted` — the child rows whose parent is in the restricted set. -- **upstream** (a restriction on the child, carried to the parent): `parent & child_restricted` — the parent rows referenced by the restricted child. +**Row relations.** For a row `r`: -Downstream and upstream are the *same* operation pointed opposite ways along the same foreign key. This is the whole engine; everything below is how the edge rule degenerates, what parts add, and how you accumulate across many edges. +- `child(r) = {c : ref_e(c) = r for some foreign key e}` — the rows that reference `r`; +- `parent(r) = {ref_e(r) : e a foreign key of r's table, ref_e(r) defined}` — the rows `r` references; +- `down(r) = child(r) ∪ inst(r)`; +- `up(r) = parent(r) ∪ inst(r)`. -### 2. The edge rule (R1, referential) +`up` is exactly the converse of `down`: `y ∈ down(x)` if and only if `x ∈ up(y)`. `parent` is the converse of `child`, and sharing an instance is symmetric. -**Base case — the foreign key is the whole primary key, not renamed, no parts.** The parent's primary key is embedded verbatim in the child's, with the same column names. A primary-key restriction on the parent is a predicate on exactly those columns, which the child also has, by the same names. So: **carry the restriction unchanged** — the identical predicate that selects the parent rows already selects the matching child rows. This is why the same primary-key restriction rides the whole diagram. +### Operations -**Complication A — secondary foreign key.** The parent's key lands in the child's *secondary* (non-primary) attributes. A raw predicate still selects the right child rows, but the restriction is no longer a statement about the child's *identity*, so it cannot be promoted to the child's primary key and ridden further. Keep it relational: project the restricted parent to its key and restrict the child by it, matched on the foreign-key columns. +A set of rows `X` is **closed under** a relation `R` when, for every row `x` in `X`, all of `R(x)` is in `X` too. The **closure** of a set `S` under `R` is the smallest set that contains `S` and is closed under `R`. It is computed by starting from `S` and adding `R`-neighbors until nothing new appears; in mathematical terms it is the *least fixed point* of that step. Every closure here is finite because the database is. -**Complication B — renamed foreign key.** The referencing columns have different names in the child. The parent's predicate names columns the child lacks. Fix, mechanically: rename the restriction's columns through the foreign key's attribute map before restricting (reverse the rename going upstream). +Let `S` be the set of rows the seed selects. Then: -> **R1 (edge rule):** propagate a restriction across a foreign-key edge by **restricting** the neighbor by the restricted table (`&`), projected/renamed onto the shared foreign-key columns. When the foreign key is the whole primary key and unrenamed, the projection is the identity and the restriction collapses to "apply the same predicate." +- `D(S)` is the closure of `S` under `down`; +- `U(S)` is the closure of `S` under `up`; +- `D1(S)` is the closure of `S` under `child` alone, with no instance step. -### 3. The group rule (R2, compositional) +| Call | Rows | +|---|---| +| `expand(seed, "down")` | `D(S)` | +| `expand(seed, "up")` | `U(S)` | +| `expand(seed, "both")` | `U(D(S))` | -Part tables add **compositional** integrity on top of referential integrity: a master and its parts are one entity, created and deleted all-or-nothing. +**Why `both` is `U(D(S))`.** An export region must be importable on its own: every foreign key in it must resolve inside it, and every instance in it must be whole. `U(D(S))` is the smallest such region that contains everything derived from `S` (T6). The other two combinations fail: -- **master -> part** needs nothing new — a part carries `-> master` in its primary key, so R1 already sweeps in all parts of a restricted master. -- **part -> master** is the new rule. A restriction landing on part rows satisfies referential integrity by touching just those rows, but leaves a fragment of an entity. So it must **lift existentially to the master** (the master is in if *any* of its parts is), and the master re-expands to **all** its parts. +- `D(U(S))` adds every descendant of every ancestor. For a seed under a shared lookup table, that is most of the database. +- `D(S) ∪ U(S)` misses the other parents of the descendants. In [S5](traversal-shapes.md#s5-parallel-foreign-keys), exporting neuron 1 this way would carry synapse (3, 1) without neuron 3. -> **R2 (group rule):** a restriction touching any part of a master's group brings the whole group — existential lift part -> master, then expand master -> all parts. +**Delete.** `Table.delete(part_integrity=...)` and its preview `Diagram.cascade()`: -R2 is a closure over the master–part grouping, which is exactly why foreign-key restrictions alone cannot express it. +| `part_integrity` | Deletes | +|---|---| +| `"cascade"` | `D(S)` | +| `"ignore"` | `D1(S)`. A part row can be deleted while its master and sibling part rows stay. | +| `"enforce"` (default) | `D1(S)`, if it is **lift-closed**: it contains `master(r)` for every part row `r` in it. Otherwise it raises **before deleting anything**, naming a part row and its master. | -### 4. The one operation: expand +By T7, `enforce` deletes `D(S)` when the group rule would add nothing to `D1(S)`, and refuses otherwise. -There is one traversal. Seed a single restricted table and grow outward by R1 + R2, accumulating reachable rows by **union** (a table is reached if reachable via any path). It is directional, `direction="down" | "up" | "both"` (default `"down"`): +**Trace.** `Diagram.trace(seed)` selects `U(S)`. -- `direction="down"` — descendants: the **delete blast radius**. This is `cascade`. -- `direction="up"` — ancestors: the **valid query sources** a `make()` may read under the reproducibility contract. This is `trace`; inside `make()`, `self.upstream` is `expand(self & key, direction="up")`. -- `direction="both"` — a referentially-consistent **export region** around the seed: everything the seed rows depend on and everything derived from them. +**`self.upstream`.** `make(key)` runs before the row for `key` is inserted, so `self & key` selects nothing, and `U(∅) = ∅` would leave `self.upstream` empty. `self.upstream` therefore starts from the parents that row will have. For each foreign key `e` of the table that lies in its primary key, `relabel_e(key)` renames `key`'s values onto the primary key of the parent `P_e` through `attr_map_e`, and -A single-seed closure is always consistent and never needs an intersection: tracing up pulls exactly the referenced ancestors, cascading down pulls exactly the dependents. Every requirement is a case of `expand`: blast radius (`down`), `make()` sources (`up`), and "all data for this entity" / consistent export (`both`). +``` +self.upstream = ⋃ over primary foreign keys e of expand(P_e & relabel_e(key), "up") +``` -**Multiple conditions do not need a second operation.** A filter over several *independent* tables (e.g. "data for `subject_id=5` **and** `method_id=5`") is still `expand`. Either the conditions share a common descendant that inherits both keys — seed that descendant with the combined condition and `expand(both)` — or they do not, in which case no table is downstream of both, there is nothing to intersect, and the result is the union of the individual expansions. A per-table "conjunction of upstream conditions" carving is a UI-layer composition of `expand`s (a Navigator concern), not a core operation. +Foreign keys outside the primary key are left out: their values are not known before the insert. By T2, `self.upstream` is `U` of the parents of the row being computed. -### 5. Why this is the whole story +!!! version-changed "Changed in 2.4" + In 2.3, `self.upstream` was `Diagram.trace(self & key)`. It returned the parents only because the engine copied the key's condition upward, which is not sound for other seeds (see [R1](#r1-the-edge-rule)). -- **It is pure reachability.** A restriction reaches a table if it reaches it via *any* foreign-key path, so convergence is always **union** — there is no intersection to reason about and no second, subtractive operation. -- **One data structure.** A diagram is a set of tables, each holding one row-set; `expand` unions reachable rows into it as it grows. -- **Materialization is a delete-time concern, not part of traversal.** Freezing a group's keys before deleting (delete runs parts-before-masters) matters only when a traversal feeds `delete`; the read-only closures never pay for it. +### Derived results -### 6. Renamed foreign keys and the seed restriction +Each result below follows from the definitions and is checked by the test suite. -Each time a restriction crosses a foreign key it must be re-expressed in the neighbor's attribute names. Renaming is the only thing that changes names across an edge, so it is the only place this needs care, and the *shape* of the seed restriction decides how. A restriction is one of three kinds: +**T1, closure.** `D`, `U`, and `D1` are *extensive* (`X(S) ⊇ S`), *monotone* (`A ⊆ B` implies `X(A) ⊆ X(B)`), and *idempotent* (`X(X(S)) = X(S)`). -- **materialized** — a dict of primary-key values, or a sequence of them (literal `attr: value` rows, e.g. `A.keys()`); -- **subquery** — a query expression (another table, possibly restricted); -- **string** — a raw SQL predicate over attribute names, e.g. `'weight > 10'`. +Every closure under a relation has these properties: the result contains `S` by construction, a larger starting set reaches at least the same rows, and a set that is already closed gains nothing from closing it again. -Only the materialized kind is *frozen literal values*; the other two are *live* (evaluated against current data). This split decides whether a renamed edge is crossed by simply relabelling or must be crossed relationally. +**T2, additivity.** `X(A ∪ B) = X(A) ∪ X(B)` for `X` = `D`, `U`, `D1`, and for `both`. -**Kind 1 — materialized (relabel fast-path).** Crossing a renamed foreign key, the neighbor's restriction is obtained by **renaming the dict's keys through the edge, values unchanged**: keep the referenced attributes (relabelled) and drop any key field that does not exist on the neighbor. Going upstream this drops the child's own identity attributes, leaving exactly the parent's key; going downstream nothing is dropped and the child's own key attributes stay unconstrained (a partial key). Renamings chain, so a key relabels edge by edge. This is exact because the renaming is pure (values and types preserved) and attribute identity across the edge is fixed by the edge's pairing, not by coincidental name matches. A sequence of dicts (e.g. `A.keys()`) is the OR of its members and relabels element by element; the walk stays symbolic — no subqueries — and, being frozen literals, is stable and delete-safe. +Each step adds the neighbors of a single row, so a row is in `X(S)` exactly when a chain of steps leads to it from one row of `S`, and that is the case for `A ∪ B` exactly when it is the case for `A` or for `B`. For `both`, `U(D(A ∪ B)) = U(D(A) ∪ D(B)) = U(D(A)) ∪ U(D(B))`. So several seeds give the union of their expansions: `expand` never needs an intersection, `d1 + d2` of the same direction is exact, and `self.upstream` is built as a union. -*Which attributes cross an edge:* an attribute propagates across an edge iff that edge **carries** it (the foreign key references it). A data attribute (`weight`) is carried by no edge and never propagates — a key containing one is really an `A & cond` case (Kind 2/3), reducible to Kind 1 by materializing to keys first, `(A & cond).keys()`. A *secondary* foreign-key attribute propagates along its own edge even though it is not part of the primary key. So the test is per edge — *does the key cover the attributes this edge carries?* — not a global "is the key primary-key-only?". +**T3, duality.** `y ∈ D({x})` if and only if `x ∈ U({y})`. -*Direction, because a foreign key is a function:* **down** (parent to children) is the preimage — a parent key relabels to a partial child key and always suffices. **Up** (child to parent) is the image — to name the referenced parent by relabelling, the key must include the parent's **full primary key** (in the child's names). When it doesn't, the relabel fast-path cannot fire and `expand` **materializes** — queries the child for the actual referenced parent keys — and continues. +A chain of `down` steps from `x` to `y`, read backwards, is a chain of `up` steps from `y` to `x`, because `up` is the converse of `down`. So deleting `x` invalidates exactly the rows whose upstream contains `x`. Because a delete lifts part rows to their master, a trace must include the parts of an ancestor master, and their ancestors ([#1481](https://github.com/datajoint/datajoint-python/issues/1481); [G5](traversal-shapes.md#g5-merge-table)). `ignore`'s `D1` is likewise dual to a trace without the instance step. -**Kinds 2 & 3 — live (restrict-then-project).** A subquery or string restriction has no literal values to relabel; cross the edge **relationally** — restrict `A` by the condition, project it onto the neighbor's referenced attributes (renamed), and restrict the neighbor. A string cannot cross as text (it may name attributes the foreign key doesn't carry, and rewriting SQL to the neighbor's names is not reliable); only its *effect* crosses, through the referenced-attribute values of the surviving `A` rows. Any live restriction becomes relabel-able (and delete-safe) the moment it is materialized to keys, `(A & r).keys()` — which is exactly what `cascade` does at plan time. +**T4, delete safety.** If the database satisfies P1, removing `D(S)` keeps P1 and removes each instance either whole or not at all (P2). `D(S)` is the smallest superset of `S` with both properties (P4). -### 7. The group rule and the relabel fast-path +`D(S)` contains every child of every row in it, so no remaining row references a removed one; and it contains the instance of every row in it, so no instance is split. Conversely, a removed set `X ⊇ S` that keeps P1 must contain the children of its rows, and one that keeps instances whole must contain their instances. Such an `X` is closed under `down` and so contains `D(S)`. -R2 needs **no new key machinery** — it is the relabel fast-path run twice, with the part-specific attribute deliberately lost in between. For a master `Session` (primary key `session_id`) and part `Session.Trial` (primary key `(session_id, trial_id)`): +**T5, read boundary.** `U(S)` is consistent on its own: every foreign key of its rows resolves inside it, and every instance in it is whole. It is the smallest superset of `S` with that property, and so the set of rows `S` depends on (P3). -- **master to part (down):** the ordinary relabel — `Session & {'session_id': 5}` becomes the partial key `{'session_id': 5}` on `Session.Trial`, selecting *every* trial of session 5. "All parts follow the master" falls straight out of the down rule. -- **part to master (up), the existential lift:** relabel-drop — `Session.Trial & {'session_id': 5, 'trial_id': 2}` keeps `session_id`, drops the part-specific `trial_id`, giving `{'session_id': 5}`. A sequence of part keys across many trials all drop to the same master key and de-duplicate, so the OR-over-siblings is free. -- **master to all parts (re-expansion):** *not* a relabel of the seed — a fresh downstream step from the recovered master key, `{'session_id': 5}` on `Session` relabels down to `{'session_id': 5}` on `Session.Trial`, which drops the `trial_id` constraint and so *widens* from "trial 2" to all trials. +The argument is T4's with `parent` in place of `child`. -**The signature: a key that reaches a part via its master carries no part-specific constraint.** The lift *narrows* the key to the master; the re-expansion *widens* it to all parts; the part-specific attribute is destroyed by the lift and cannot be recovered. That loss *is* compositional atomicity in key terms — the whole part-group comes along precisely because the returning key no longer distinguishes one part from its siblings. +**T6, `both`.** `U(D(S))` contains `D(S)`, is closed under `up`, and so is consistent on its own and contains every instance it touches. It is the smallest set with these properties that contains `D(S)`. `both` is extensive and monotone but not idempotent. -Corollaries: a materialized-key seed stays materialized through the whole part -> master -> parts round trip, so it is delete-safe for free (the delete-order materialization only ever fires for *live* seeds); and the same mechanism serves both directions — the mutating part->master cascade (`expand` down feeding a delete) and the upstream `trace` (`expand` up) that surfaces an ancestor master's parts. +`U(D(S))` is `U` of `D(S)`, so T5 applies to it with `D(S)` as the seed. For non-idempotence, take a table `A` with two child tables `B` and `C`, and a row `a` with one child `b` in `B` and one child `c` in `C`. With `S = {b}`, `both(S) = U({b}) = {a, b}`, and `both({a, b}) = U({a, b, c}) = {a, b, c}`: the second application picks up `a`'s other descendant. -### Summary +**T7, enforce.** Under A1, `D1(S)` is lift-closed if and only if `D1(S) = D(S)`. -| direction | operation | serves | -|---|---|---| -| `down` | `expand(seed, "down")` = `cascade` | delete blast radius | -| `up` | `expand(seed, "up")` = `trace` | `make()` sources (`self.upstream`) | -| `both` | `expand(seed, "both")` | consistent export region / "all data for this entity" | +Under A1, every part row of an instance can be reached from its master row by `child` steps, through `-> master` or through sibling parts that carry the same key. So if `D1(S)` contains the master of each of its part rows, it contains their whole instances, is closed under `down`, and equals `D(S)`. Conversely, `D(S)` is closed under `inst`, so if `D1(S) = D(S)` it is lift-closed. `enforce` therefore deletes only when the group rule would add nothing. + +**T8, ordering.** Under A1, `topo_sort` succeeds if and only if the group quotient `Q` is acyclic. The **group quotient** `Q` is the table-level foreign-key graph with each master and its parts merged into a single node. + +`topo_sort` sorts the table graph after adding, for each part, an edge from each of the part's parents outside its group to its master. Under A1 the master reaches each of its parts, so a cycle in `Q` becomes a cycle in that graph. Conversely, a cycle in that graph cannot stay inside one group, because foreign keys between tables never form a cycle, so it maps to a cycle in `Q`. Declaring a part that would close a cycle in `Q` is refused ([X2](traversal-shapes.md#x2-group-cycle-refused-at-declaration)), so `Q` stays acyclic. + +### Computation + +The engine computes the rows above as restrictions on tables, without reading the rows, except where noted. + +#### R1, the edge rule + +**Down.** Crossing a foreign key `e` from a restricted parent `P` to the child `C`, the child is restricted by the restricted parent, projected onto `P`'s primary key, with each key attribute renamed to its name in `C` through `attr_map_e`: + +``` +C & P_restricted.proj() +``` + +The parent's condition may instead be **copied** to the child unchanged, but only when the attributes it names are among `e`'s attributes, `e` is not renamed, and none of `e`'s columns is nullable. Then each child row holds the same values under the same names as the parent row it references, and it references one, so the copied condition selects exactly the children. [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column) shows the nullable case, [S4](traversal-shapes.md#s4-renamed-foreign-key) the renamed case, and [S7](traversal-shapes.md#s7-diamond-reached-through-a-secondary-foreign-key) a condition on an attribute the child holds through a different parent. + +A copy is a shortcut: it must select the same rows as the restriction, so the condition must also mean the same thing on the child as on the parent. Two kinds of condition can change meaning when moved, and are not copied: + +- a dict with keys that are not attributes of the parent: they are ignored on the parent but can match attributes of the child; +- a query expression that shares more attributes with the child than with the parent: a restriction by a query expression matches on all shared attributes. Copied to `Session`, the seed condition of [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant), `Session & "quality = 0"`, would match on every attribute of `Session` and select only the sessions of quality 0. Copied the same way, the seed `Subject - (Session & "quality = 0")` would select every session of another quality, including sessions of subjects that are not deleted. + +**Parallel foreign keys** combine with OR: a child row is reached if any of its foreign keys references a restricted row. + +**Up.** Crossing `e` from a restricted child `C` to the parent `P` is always the restriction + +``` +P & C_restricted.proj() +``` + +This selects the parent rows referenced by at least one restricted child row. It is never a copy: a condition on the child describes child rows, and copied to the parent it would select parents that no restricted child references ([S2](traversal-shapes.md#s2-secondary-foreign-key), [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column)). + +The one shortcut upward applies to keys materialized from the child table itself (for example `(C & cond).keys()`) that include every column of `e`: these may be relabeled through `attr_map_e`. They name existing child rows, so the relabeled keys name exactly the referenced parents. A user-supplied key dict may name rows that do not exist, so it crosses upward by restriction too ([K5](traversal-shapes.md#k5-empty-seeds)). + +#### R2, the group rule + +For a master `M` and its part `Pi`, with `R(T)` the restriction on table `T`: + +- **lift:** `M & R(Pi).proj(*M.primary_key)` — the masters of the restricted part rows. A1 puts the master's key in every part row, so the lift is one step on that key, with no walk along the foreign keys between parts ([G2](traversal-shapes.md#g2-part-of-part-with-two-chains)). Another foreign key from the part to its master is not used for the lift ([G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master)). +- **re-expand:** `Pi & R(M)` — every part row of the restricted masters. + +The lift is existential: a master joins only if at least one of its part rows did ([G1](traversal-shapes.md#g1-part-to-master-lift)). + +#### One pass over the group quotient + +Because `Q` is acyclic (T8), the engine visits each node of `Q` once: in `topo_sort` order for `down`, and in reverse order for `up`. + +- A **plain table** `T` gets `R(T)` = its seed ∪ R1 from each restricted neighbor on the incoming side (its parents for `down`, its children for `up`). +- A **group** with master `M` gets `R(M)` = the seed on `M` ∪ R1 into `M` ∪ lift(R1 into each part ∪ the seed on each part). Then each part `Pi` gets `Pi & R(M)`. + +`up` is the same pass with every edge reversed. The pass is exact because every neighbor on the incoming side is final before it is used (`Q` is acyclic), and because a foreign key inside a group maps the master's key to itself, so steps inside a group never leave the instances already in `R(M)`. The next subsection covers the groups where that fails. + +#### Groups that iterate + +A foreign key inside a group that renames the master's key links different instances of the group. In [G4](traversal-shapes.md#g4-renamed-sibling-foreign-key-chained-across-three-instances), a `Carry` row of run 2 references a `Step` row of run 1; in [G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master), a part references another row of its own master. The engine detects such a foreign key from its `attr_map`. For that group only, it evaluates the lifted master keys and repeats the group step until no new key appears. This is the only place where a traversal reads rows before a delete. + +#### Seeds + +- A seed is one table with any restriction. Projections, joins, aggregations, and unions are refused as seeds. The restriction itself may use them, for example a subquery on a descendant ([K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)) or on the seed's own table ([K2](traversal-shapes.md#k2-self-referencing-seed)). +- A `dj.Top` seed is first replaced by the primary keys it selects ([K4](traversal-shapes.md#k4-djtop-seed)). +- A dict that names none of the table's attributes means all rows, as in any restriction ([Query Algebra §2.4](query-algebra.md#24-dictionary-conditions)). As a seed, it raises a warning ([K6](traversal-shapes.md#k6-unrestricted-seed)). +- A materialized seed (a dict, a list of dicts, or `.keys()`) may cross an edge by **relabeling**: keeping only the keys the foreign key carries, renamed through `attr_map`. Relabeling is used where it selects the same rows as R1: downward, when the foreign key has no nullable column or the keys name all its columns; upward, only for keys read from the child table that contain all its columns. A live seed — a string or a query expression — crosses by restriction. Both give the same rows ([K3](traversal-shapes.md#k3-seed-kinds-give-the-same-rows)). +- The attributes a string condition names are read from its text outside any subquery. + +#### The plan + +The engine first builds a **plan**: an intermediate form, not SQL. Its nodes are `Seed`, `Restrict(src, attr_map)`, `Lift(part)`, `Reexpand(master)`, `Or`, and `Literal`. A node refers to the restrictions it uses by a handle to another node; compiled SQL is never copied from one restriction into another. The plan compiles to query expressions for the database. It can also be evaluated against rows held in memory, which is how the test suite compares it with the definitions above. + +#### Materialization happens only at delete time + +Traversal and preview (`counts()`, iteration, `d[T]`) stay symbolic, apart from [groups that iterate](#groups-that-iterate). Before the first DELETE, the engine **materializes** a node, replacing its restriction by the literal keys it selects, when one of the node's local terms reads a table that is deleted no later than the node itself. A *local term* is the node's seed or a lift into it. The terms a node inherits through R1 from a parent are safe: `Table.delete()` deletes in reverse topological order, so a parent still holds its rows when its children are deleted. + +In practice: + +- every lift is materialized, because a part is deleted before its master; +- a seed is materialized unless its restriction is literal keys, because a string or query-expression restriction is SQL text whose reads cannot be inspected ([K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant), [K2](traversal-shapes.md#k2-self-referencing-seed)); +- large key sets are applied in chunks. + +Without this, a restriction that reads a table emptied earlier in the same delete would match nothing, and its rows would stay behind ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)). + +#### Concurrency + +- Part rows are deleted as `Pi & R(M)`, with `M` materialized. A part row inserted by another session after that DELETE makes the master's DELETE fail on its foreign key (`ON DELETE RESTRICT`), so the transaction rolls back and the part row is never left without its master. +- The foreign-key error message mentions concurrent inserts. +- `enforce` runs the `down` plan once its check passes. By T7, its rows are `D1(S)`. +- The table-level check after the delete stays as a cheap backstop. + +### Expanded diagrams + +A diagram returned by `expand()`, `cascade()`, or `trace()` is an **expanded diagram**. It has a direction, and it obeys the following contract. + +- **Nodes.** The node set is the *table-level reach* of the seed's table: the tables reached by following foreign keys in the direction of travel, together with the master and all parts of every group that has a reached member. (For `D1`, the group step is left out.) For `both`, it is the upward reach of the downward reach. A node may hold zero rows. +- **`d[T]`** returns `T` restricted to its rows, in every direction, and raises `DataJointError` if `T` is not a node. +- **`counts()`** works, and **iteration** runs in topological order. +- **Copying** a diagram, `dj.Diagram(d)`, keeps its direction. +- **`d1 + d2`** of two diagrams with the same direction holds, for each table, the union of their rows (T2). Adding diagrams of different directions raises. +- **`-`, `*`, `+ n`, and `- n`** raise. +- **`prune()`** raises for `down` and is allowed for `up` and `both`. + +!!! version-changed "Changed in 2.4" + In 2.3, copying a diagram dropped its direction, and `d1 + d2` silently dropped the right operand's restrictions. + +### Retiring `restrict` + +`Diagram.restrict()` is deprecated in 2.4: it emits a `DeprecationWarning`, its behavior is frozen, and it is removed in 3.0. It intersected the conditions that reached a table (AND), which can split an instance, keeping a master with only some of its parts ([#1501](https://github.com/datajoint/datajoint-python/issues/1501)). `expand` takes unions instead, so its results are larger: they hold everything the seed reaches, and never part of an instance. + +| Use of `restrict` | Replacement | +|---|---| +| One condition, for export | `dj.Diagram.expand(Session & cond, "both")` | +| One condition, for impact | `dj.Diagram.expand(Session & cond, "down")` | +| Several conditions that share a descendant | Seed the descendant with all of them, e.g. `dj.Diagram.expand(Session & (Subject & cond_a) & cond_b, "both")` | +| Per-table AND across several seeds | The recipe below | + +The recipe intersects, table by table, the rows that separate `down` expansions reach: + +```python +seeds = [Subject & {"species": "mouse"}, Session & "session_date > '2024-01-01'"] +carved = {} +for diagram in (dj.Diagram.expand(seed, "down") for seed in seeds): + for table in diagram: # restricted FreeTable, parents first + name = table.full_table_name + carved[name] = carved[name] & table.proj() if name in carved else table +``` + +Each `carved[name]` holds the rows that every seed reaching that table reaches. Unlike `restrict`, a single seed's paths combine by union. The result is not closed: it can split an instance and need not be importable on its own, so use it for inspection, and seed a common descendant for export. -One operation, one data structure, two rules — R1 (edge restriction) and R2 (group). `cascade` and `trace` are the downstream and upstream cases of `expand`; a filter over several independent tables composes `expand`s rather than adding a new operation. +The [Traversal Shape Catalog](traversal-shapes.md) gives the exact rows for each schema shape the specification distinguishes. --- @@ -569,7 +709,7 @@ combined = dj.Diagram.from_sequence([schema1, schema2, schema3]) ## Dependencies -Operational methods (`cascade`, `restrict`, `counts`, `prune`) use `networkx`, which is always installed as a core dependency. +Operational methods (`expand`, `cascade`, `trace`, `counts`, `prune`) use `networkx`, which is always installed as a core dependency. Diagram **visualization** requires optional dependencies: @@ -584,6 +724,10 @@ If visualization dependencies are missing, `dj.Diagram` displays a warning and p ## See Also - [How to Read Diagrams](../../how-to/read-diagrams.ipynb) +- [Traversal Shape Catalog](traversal-shapes.md) — Worked examples of `expand`, one per schema shape +- [Cascade Specification](cascade.md) — The delete case of `expand` +- [Upstream Trace Specification](trace.md) — The upstream case of `expand` and `self.upstream` +- [Master-Part](master-part.md) — Master-part groups - [Delete Data](../../how-to/delete-data.md) — Cascade inspection and delete workflow - [What's New in 2.2](../../about/whats-new-22.md) — Motivation and design - [Data Manipulation](data-manipulation.md) — Insert, update, delete specification diff --git a/src/reference/specs/index.md b/src/reference/specs/index.md index ac7d6bb9..baba6662 100644 --- a/src/reference/specs/index.md +++ b/src/reference/specs/index.md @@ -98,6 +98,7 @@ Each specification follows a consistent structure: | [Primary Keys](primary-keys.md) | Table Declaration | [Design Primary Keys](../../how-to/design-primary-keys.md/) | [Entity Integrity](../../explanation/entity-integrity.md/) | | [Fetch API](fetch-api.md) | Query Operators | [Fetch Results](../../how-to/fetch-results.md/) | — | | [Diagram](diagram.md) | Table Declaration | [Read Diagrams](../../how-to/read-diagrams.ipynb/) | — | +| [Traversal Shape Catalog](traversal-shapes.md) | Diagram, Master-Part | [Delete Data](../../how-to/delete-data.md/) | — | **Key concepts:** Restriction (`&`, `-`), projection (`.proj()`), join (`*`), aggregation (`.aggr()`), union, universal set (`U()`), attribute lineage, schema visualization diff --git a/src/reference/specs/traversal-shapes.md b/src/reference/specs/traversal-shapes.md new file mode 100644 index 00000000..e5660e00 --- /dev/null +++ b/src/reference/specs/traversal-shapes.md @@ -0,0 +1,804 @@ +# Traversal Shape Catalog + +Worked examples of `Diagram.expand`, one per schema shape that the [Traversal specification](diagram.md#traversal-specification) distinguishes. + +!!! version-added "New in 2.4" + `Diagram.expand` and the behavior shown here are new in DataJoint 2.4. + +## How to read this catalog + +Each shape has a tiny schema, a few rows, one or more seeds, and the exact rows `expand` returns. The rows follow from the definitions of `D` and `U` in [Operations](diagram.md#operations); a sentence after each shape says why. The test suite mirrors these shapes one to one, under the same IDs. + +- **Directions.** *down* is `dj.Diagram.expand(seed, "down")`, which equals `Diagram.cascade(seed, part_integrity="cascade")`. *up* is `expand(seed, "up")`, which equals `Diagram.trace(seed)`. *both* is `expand(seed, "both")`. +- **Schemas.** Each shape is its own schema, declared with `schema = dj.Schema(...)`; table names repeat across shapes. Tables are `dj.Manual` unless the shape says otherwise. +- **Rows** are written as primary-key tuples, in the order of the primary key. When a column outside the primary key matters, it follows a semicolon: `(1, 2; NULL)` is a row with primary key `(1, 2)` whose foreign-key column is NULL. +- **Results** list every table in the diagram's node set (the [table-level reach](diagram.md#expanded-diagrams)). *none* means the table is a node with zero rows. + +| ID | Shape | What it pins down | +|---|---|---| +| [S1](#s1-identity-foreign-key-in-the-primary-key) | Identity foreign key in the primary key | The base case: the condition is copied | +| [S2](#s2-secondary-foreign-key) | Secondary foreign key | Upward crossing is a restriction | +| [S3](#s3-nullable-foreign-key-sharing-a-primary-key-column) | Nullable foreign key sharing a primary-key column | A foreign key with a NULL column references nothing | +| [S4](#s4-renamed-foreign-key) | Renamed foreign key | Crossing through `attr_map`, not by name | +| [S5](#s5-parallel-foreign-keys) | Parallel foreign keys | Separate edges combine with OR | +| [S6](#s6-diamond-with-a-shared-key) | Diamond with a shared key | Paths combine by union | +| [S7](#s7-diamond-reached-through-a-secondary-foreign-key) | Diamond reached through a secondary foreign key | A row reached only through its secondary reference | +| [G1](#g1-part-to-master-lift) | Part-to-master lift | The lift is existential | +| [G2](#g2-part-of-part-with-two-chains) | Part-of-part with two chains | The lift is one step on the master key | +| [G3](#g3-a-second-renamed-foreign-key-to-its-own-master) | A second, renamed foreign key to its own master | Only the owning reference lifts | +| [G4](#g4-renamed-sibling-foreign-key-chained-across-three-instances) | Renamed sibling foreign key, chained across three instances | The group step iterates | +| [G5](#g5-merge-table) | Merge table | A trace reaches the parts of an ancestor master | +| [G6](#g6-a-part-violating-a1) | A part violating A1 | Traversed as a plain table | +| [K1](#k1-seed-restricted-by-a-descendant) | Seed restricted by a descendant | Seeds are materialized before deleting | +| [K2](#k2-self-referencing-seed) | Self-referencing seed | Same, for a seed that reads its own table | +| [K3](#k3-seed-kinds-give-the-same-rows) | Seed kinds give the same rows | Dict, list, string, subquery, `AndList`, `Not` | +| [K4](#k4-djtop-seed) | `dj.Top` seed | `Top` is replaced by its keys | +| [K5](#k5-empty-seeds) | Empty seeds | Nothing in, nothing out | +| [K6](#k6-unrestricted-seed) | Unrestricted seed | All rows, and a warning for a dict that names no attribute | +| [X1](#x1-unloaded-downstream-schema) | Unloaded downstream schema | Schemas are discovered through foreign keys | +| [X2](#x2-group-cycle-refused-at-declaration) | Group cycle refused at declaration | The group quotient stays acyclic | + +--- + +## Foreign-key shapes + +### S1. Identity foreign key in the primary key + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Session` | `subject_id, session_id` | (1, 1), (1, 2), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & {"subject_id": 1}` | down | `Subject`: 1
`Session`: (1, 1), (1, 2) | +| `Session & {"session_id": 1}` | up | `Session`: (1, 1), (2, 1)
`Subject`: 1, 2 | + +**Why.** The foreign key embeds the parent's whole key under the same names, in the child's primary key, so the condition `subject_id = 1` is copied to `Session` unchanged. Upward, `Subject` is restricted by the restricted sessions, which reference subjects 1 and 2. + +### S2. Secondary foreign key + +```python +@schema +class Rig(dj.Lookup): + definition = """ + rig_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + --- + -> Rig + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Rig` | `rig_id` | 1, 2, 3 | +| `Session` | `session_id; rig_id` | (1; 1), (2; 1), (3; 2) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Rig & {"rig_id": 1}` | down | `Rig`: 1
`Session`: 1, 2 | +| `Session & {"session_id": 3}` | up | `Session`: 3
`Rig`: 2 | + +**Why.** Downward, the condition names only `rig_id`, the foreign key's own non-null attribute, so it is copied. Upward, the seed's condition names `session_id`, which the foreign key does not carry: `Rig` is restricted by the restricted session's `rig_id`, giving rig 2. Relabeling the dict would drop `session_id` and select every rig. + +### S3. Nullable foreign key sharing a primary-key column + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +@schema +class Surgery(dj.Manual): + definition = """ + -> Subject + surgery_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + --- + -> [nullable] Surgery + """ +``` + +`Session`'s foreign key to `Surgery` is `(subject_id, surgery_id)`. It shares `subject_id` with `Session`'s primary key, so only `surgery_id` can be NULL. + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Surgery` | `subject_id, surgery_id` | (1, 1), (1, 2), (2, 1) | +| `Session` | `subject_id, session_id; surgery_id` | (1, 1; 1), (1, 2; NULL), (1, 3; 2), (2, 1; 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Surgery & {"subject_id": 1}` | down | `Surgery`: (1, 1), (1, 2)
`Session`: (1, 1), (1, 3) | +| `Session & {"subject_id": 1, "session_id": 2}` | up | `Session`: (1, 2)
`Subject`: 1
`Surgery`: none | + +**Why.** Session (1, 2) has `subject_id` set and `surgery_id` NULL, so it references no surgery (A2). Downward, the foreign key has a nullable column, so `subject_id = 1` is not copied; a copy would also delete session (1, 2) ([#1590](https://github.com/datajoint/datajoint-python/issues/1590)). Upward, session (1, 2) reaches its subject but no surgery. + +### S4. Renamed foreign key + +```python +@schema +class Neuron(dj.Manual): + definition = """ + neuron_id : int32 + """ + + +@schema +class Input(dj.Manual): + definition = """ + -> Neuron.proj(source_id="neuron_id") + input_id : int32 + --- + neuron_id : int32 # a local label, not a foreign key + """ +``` + +`Input` holds the referenced neuron as `source_id`, and also has an unrelated attribute that happens to be named `neuron_id`. + +| Table | Columns | Rows | +|---|---|---| +| `Neuron` | `neuron_id` | 1, 2, 3 | +| `Input` | `source_id, input_id; neuron_id` | (1, 1; 2), (1, 2; 3), (2, 1; 1), (3, 1; 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Neuron & {"neuron_id": 1}` | down | `Neuron`: 1
`Input`: (1, 1), (1, 2) | +| `Input & {"source_id": 2, "input_id": 1}` | up | `Input`: (2, 1)
`Neuron`: 2 | + +**Why.** A renamed foreign key is crossed through its `attr_map` (`source_id` ↔ `neuron_id`), never by attribute name. Copying `neuron_id = 1` to `Input` would match the unrelated label and select inputs (2, 1) and (3, 1). + +### S5. Parallel foreign keys + +```python +@schema +class Neuron(dj.Manual): + definition = """ + neuron_id : int32 + """ + + +@schema +class Synapse(dj.Manual): + definition = """ + -> Neuron + -> Neuron.proj(target_id="neuron_id") + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Neuron` | `neuron_id` | 1, 2, 3 | +| `Synapse` | `neuron_id, target_id` | (1, 2), (2, 3), (3, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Neuron & {"neuron_id": 1}` | down | `Neuron`: 1
`Synapse`: (1, 2), (3, 1) | +| `Synapse & {"neuron_id": 1, "target_id": 2}` | up | `Synapse`: (1, 2)
`Neuron`: 1, 2 | +| `Neuron & {"neuron_id": 1}` | both | `Neuron`: 1, 2, 3
`Synapse`: (1, 2), (3, 1) | + +**Why.** The two foreign keys are separate edges, and a synapse is reached if either one references neuron 1; combining them with AND would select only synapses from neuron 1 to itself, of which there are none. *both* adds neurons 2 and 3, the other ends of the exported synapses; `D(S) ∪ U(S)` would leave them out. + +### S6. Diamond with a shared key + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + """ + + +@schema +class Implant(dj.Manual): + definition = """ + -> Subject + implant_id : int32 + """ + + +@schema +class Recording(dj.Manual): + definition = """ + -> Session + -> Implant + """ +``` + +`Recording` inherits `subject_id` through both parents, as one shared column. + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Session` | `subject_id, session_id` | (1, 1), (1, 2), (2, 1) | +| `Implant` | `subject_id, implant_id` | (1, 1), (1, 2), (2, 1) | +| `Recording` | `subject_id, session_id, implant_id` | (1, 1, 1), (1, 2, 2), (2, 1, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & {"subject_id": 1}` | down | `Subject`: 1
`Session`: (1, 1), (1, 2)
`Implant`: (1, 1), (1, 2)
`Recording`: (1, 1, 1), (1, 2, 2) | +| `Implant & {"subject_id": 1, "implant_id": 1}` | down | `Implant`: (1, 1)
`Recording`: (1, 1, 1) | +| `Implant & {"subject_id": 1, "implant_id": 1}` | both | `Implant`: (1, 1)
`Recording`: (1, 1, 1)
`Subject`: 1
`Session`: (1, 1) | +| `Recording & {"subject_id": 1, "session_id": 2, "implant_id": 2}` | up | `Recording`: (1, 2, 2)
`Session`: (1, 2)
`Implant`: (1, 2)
`Subject`: 1 | + +**Why.** From `Subject`, both paths carry the same `subject_id`, and their union is the result; nothing is intersected. *both* from the implant adds session (1, 1), the other parent of the exported recording, and the subject. + +### S7. Diamond reached through a secondary foreign key + +```python +@schema +class Lab(dj.Manual): + definition = """ + lab_id : int32 + """ + + +@schema +class Rig(dj.Manual): + definition = """ + rig_id : int32 + --- + -> Lab + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Lab + session_id : int32 + --- + -> Rig + """ +``` + +A session belongs to a lab through its primary key and uses a rig, which may belong to another lab, through a secondary foreign key. `Session`'s own `lab_id` and its rig's `lab_id` are independent. + +| Table | Columns | Rows | +|---|---|---| +| `Lab` | `lab_id` | 1, 2 | +| `Rig` | `rig_id; lab_id` | (10; 1), (20; 2) | +| `Session` | `lab_id, session_id; rig_id` | (1, 1; 20), (2, 1; 10), (2, 2; 20) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Lab & {"lab_id": 1}` | down | `Lab`: 1
`Rig`: 10
`Session`: (1, 1), (2, 1) | +| `Session & {"lab_id": 2, "session_id": 1}` | up | `Session`: (2, 1)
`Rig`: 10
`Lab`: 1, 2 | +| `Rig & {"rig_id": 10}` | both | `Rig`: 10
`Session`: (2, 1)
`Lab`: 1, 2 | + +**Why.** Session (2, 1) belongs to lab 2 but uses rig 10 of lab 1, so it reaches the seed only through `Rig`. `Rig`'s condition, `lab_id = 1`, names an attribute that `Session`'s foreign key to `Rig` does not carry, so it is not copied; `Session` is restricted by the restricted rigs' `rig_id`. Copying it would match sessions on their own `lab_id` and miss session (2, 1), and the DELETE on `Rig` would then fail on its foreign key. + +--- + +## Group shapes + +### G1. Part-to-master lift + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Trial(dj.Part): + definition = """ + -> master + trial_id : int32 + """ + + +@schema +class Review(dj.Manual): + definition = """ + -> Session + review_id : int32 + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Session` | `session_id` | 1, 2, 3 | +| `Session.Trial` | `session_id, trial_id` | (1, 1), (1, 2), (2, 1), (3, 1) | +| `Review` | `session_id, review_id` | (1, 1), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session.Trial & {"trial_id": 2}` | down | `Session.Trial`: (1, 1), (1, 2)
`Session`: 1
`Review`: (1, 1) | +| `Session.Trial & {"trial_id": 2}` | up | `Session.Trial`: (1, 1), (1, 2)
`Session`: 1 | + +**Why.** The lift selects the sessions that have a matching trial, `Session & (Session.Trial & {"trial_id": 2})`: only session 1. Sessions 2 and 3, and review (2, 1), are untouched. The old upward copy of `trial_id = 2` named no attribute of `Session` and selected every session ([#1591](https://github.com/datajoint/datajoint-python/issues/1591)). Upward, the instance is read whole, so trial (1, 1) comes with trial (1, 2). + +The same seed under the other delete modes: + +| `part_integrity` | Result | +|---|---| +| `"cascade"` | Deletes the *down* rows above | +| `"ignore"` | Deletes trial (1, 2) only | +| `"enforce"` | Raises before deleting anything: trial (1, 2) would be deleted without session 1 | + +### G2. Part-of-part with two chains + +```python +@schema +class Ext(dj.Manual): + definition = """ + ext_id : int32 + """ + + +@schema +class Master(dj.Manual): + definition = """ + master_id : int32 + """ + + class Q(dj.Part): + definition = """ + -> master + q_id : int32 + --- + -> Ext + """ + + class P(dj.Part): + definition = """ + -> master + p_id : int32 + --- + -> Master.Q + -> Ext + """ +``` + +`Master.P` is reachable from `Master` through two chains: directly, and through `Master.Q`. Each part also references `Ext`. + +| Table | Columns | Rows | +|---|---|---| +| `Ext` | `ext_id` | 1, 2 | +| `Master` | `master_id` | 1, 2, 3 | +| `Master.Q` | `master_id, q_id; ext_id` | (1, 10; 1), (2, 20; 2), (3, 30; 2) | +| `Master.P` | `master_id, p_id; q_id, ext_id` | (1, 100; 10, 2), (2, 200; 20, 1), (3, 300; 30, 2) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Ext & {"ext_id": 1}` | down | `Ext`: 1
`Master.Q`: (1, 10), (2, 20)
`Master.P`: (1, 100), (2, 200)
`Master`: 1, 2 | +| `Master.P & {"master_id": 1, "p_id": 100}` | up | `Master.P`: (1, 100)
`Master`: 1
`Master.Q`: (1, 10)
`Ext`: 1, 2 | + +**Why.** Downward, `Ext` 1 reaches Q (1, 10) and P (2, 200), which lift to masters 1 and 2; each master brings its whole instance, and master 3 is untouched. The lift is one step on `master_id`, which every part row carries, so the second chain needs no separate walk. Upward, the instance of P (1, 100) includes Q (1, 10), and the two parts reference `Ext` 1 and 2. + +### G3. A second, renamed foreign key to its own master + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Link(dj.Part): + definition = """ + -> master + link_id : int32 + --- + -> master.proj(other_session="session_id") + """ +``` + +`Session.Link` belongs to its session through `-> master` (the owning reference, A1) and points to another session through a renamed foreign key. + +| Table | Columns | Rows | +|---|---|---| +| `Session` | `session_id` | 1, 2, 3 | +| `Session.Link` | `session_id, link_id; other_session` | (1, 1; 2), (3, 1; 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session.Link & {"session_id": 1}` | down | `Session.Link`: (1, 1), (3, 1)
`Session`: 1, 3 | +| `Session.Link & {"session_id": 1}` | up | `Session.Link`: (1, 1)
`Session`: 1, 2 | + +**Why.** Link (1, 1) lifts through its owning reference to session 1, not to session 2. Deleting session 1 then deletes link (3, 1), which points to it, and so the rest of instance 3. Upward, link (1, 1) reads session 2, a parent, but not link (3, 1), a child. The renamed foreign key links one instance to another, so this group is [iterated](diagram.md#groups-that-iterate). + +With `part_integrity="enforce"`, `(Session & {"session_id": 1}).delete()` raises: it would delete link (3, 1) without session 3. + +### G4. Renamed sibling foreign key, chained across three instances + +```python +@schema +class Run(dj.Manual): + definition = """ + run_id : int32 + """ + + class Step(dj.Part): + definition = """ + -> master + step_id : int32 + """ + + class Carry(dj.Part): + definition = """ + -> master + carry_id : int32 + --- + -> Run.Step.proj(prev_run="run_id", prev_step="step_id") + """ +``` + +A `Carry` row of one run references a `Step` row of an earlier run. + +| Table | Columns | Rows | +|---|---|---| +| `Run` | `run_id` | 1, 2, 3, 4 | +| `Run.Step` | `run_id, step_id` | (1, 1), (2, 1), (3, 1), (4, 1) | +| `Run.Carry` | `run_id, carry_id; prev_run, prev_step` | (2, 1; 1, 1), (3, 1; 2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Run & {"run_id": 1}` | down | `Run`: 1, 2, 3
`Run.Step`: (1, 1), (2, 1), (3, 1)
`Run.Carry`: (2, 1), (3, 1) | +| `Run & {"run_id": 2}` | down | `Run`: 2, 3
`Run.Step`: (2, 1), (3, 1)
`Run.Carry`: (2, 1), (3, 1) | +| `Run & {"run_id": 2}` | up | `Run`: 1, 2
`Run.Step`: (1, 1), (2, 1)
`Run.Carry`: (2, 1) | +| `Run & {"run_id": 2}` | both | `Run`: 1, 2, 3
`Run.Step`: (1, 1), (2, 1), (3, 1)
`Run.Carry`: (2, 1), (3, 1) | + +**Why.** Deleting run 1 deletes step (1, 1), so carry (2, 1), which references it, and so the rest of run 2; that reaches carry (3, 1) and run 3. Run 4 is untouched. One pass over the group would stop at run 1. The engine detects the renamed key and repeats the group step: the master keys grow as {1}, {1, 2}, {1, 2, 3}, and a fourth round adds nothing. Upward the chain runs the other way, from run 2 back to run 1. + +### G5. Merge table + +```python +@schema +class EphysSource(dj.Manual): + definition = """ + ephys_id : int32 + """ + + +@schema +class ImagingSource(dj.Manual): + definition = """ + imaging_id : int32 + """ + + +@schema +class Recording(dj.Manual): + definition = """ + recording_id : int32 + """ + + class Ephys(dj.Part): + definition = """ + -> master + --- + -> EphysSource + """ + + class Imaging(dj.Part): + definition = """ + -> master + --- + -> ImagingSource + """ + + +@schema +class Analysis(dj.Manual): + definition = """ + -> Recording + """ +``` + +`Recording` merges two upstream sources: each recording has one part row, which names its source. + +| Table | Columns | Rows | +|---|---|---| +| `EphysSource` | `ephys_id` | 1, 2 | +| `ImagingSource` | `imaging_id` | 1 | +| `Recording` | `recording_id` | 1, 2, 3 | +| `Recording.Ephys` | `recording_id; ephys_id` | (1; 1), (2; 2) | +| `Recording.Imaging` | `recording_id; imaging_id` | (3; 1) | +| `Analysis` | `recording_id` | 1, 2, 3 | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Analysis & {"recording_id": 1}` | up | `Analysis`: 1
`Recording`: 1
`Recording.Ephys`: 1
`Recording.Imaging`: none
`EphysSource`: 1
`ImagingSource`: none | +| `EphysSource & {"ephys_id": 2}` | down | `EphysSource`: 2
`Recording.Ephys`: 2
`Recording`: 2
`Recording.Imaging`: none
`Analysis`: 2 | + +```python +trace = dj.Diagram.trace(Analysis & {"recording_id": 1}) +trace[EphysSource] # ephys source 1 +trace[ImagingSource] # empty: a node with zero rows, not an error +``` + +**Why.** Upward, recording 1 brings its whole instance. Its `Ephys` part row references the source that produced the recording, so `trace[EphysSource]` resolves ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). Downward, deleting ephys source 2 deletes the part row that names it, so the whole of recording 2, and its analysis. With `part_integrity="enforce"` that delete raises, because the part row would go without its master. + +### G6. A part violating A1 + +```python +@schema +class Probe(dj.Manual): + definition = """ + probe_id : int32 + """ + + class Channel(dj.Part): + definition = """ + -> master.proj(host_id="probe_id") + channel_id : int32 + """ +``` + +`Probe.Channel` references its master only under a renamed key, so its heading does not contain `probe_id` and it violates A1. Declaring it raises a warning, and it is traversed as a plain table: no lift and no instance. + +| Table | Columns | Rows | +|---|---|---| +| `Probe` | `probe_id` | 1, 2 | +| `Probe.Channel` | `host_id, channel_id` | (1, 1), (1, 2), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Probe.Channel & {"host_id": 1, "channel_id": 1}` | down | `Probe.Channel`: (1, 1) | +| `Probe.Channel & {"host_id": 1, "channel_id": 1}` | up | `Probe.Channel`: (1, 1)
`Probe`: 1 | +| `Probe & {"probe_id": 1}` | down | `Probe`: 1
`Probe.Channel`: (1, 1), (1, 2) | + +**Why.** As a plain table, the channel reaches its probe only as a parent: downward nothing lifts, and upward channel (1, 2) is not part of channel (1, 1)'s instance. Deleting the probe still deletes its channels, as children. + +--- + +## Seed shapes + +K1 to K6 share one schema: + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + --- + species : varchar(16) + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + --- + quality : int32 + """ + + +@schema +class Trial(dj.Manual): + definition = """ + -> Session + trial_id : int32 + """ + + +@schema +class Flag(dj.Manual): + definition = """ + flag_id : int32 + --- + -> [nullable] Subject + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id; species` | (1; mouse), (2; rat), (3; mouse), (4; rat) | +| `Session` | `subject_id, session_id; quality` | (1, 1; 0), (1, 2; 1), (2, 1; 1), (2, 2; 0), (3, 1; 1) | +| `Trial` | `subject_id, session_id, trial_id` | (1, 1, 1), (1, 2, 1), (2, 1, 1), (2, 2, 1), (3, 1, 1) | +| `Flag` | `flag_id; subject_id` | (1; 1), (2; 3) | + +Subject 4 has no sessions. + +### K1. Seed restricted by a descendant + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & (Session & "quality = 0")` | down | `Subject`: 1, 2
`Session`: (1, 1), (1, 2), (2, 1), (2, 2)
`Trial`: (1, 1, 1), (1, 2, 1), (2, 1, 1), (2, 2, 1)
`Flag`: 1 | +| `Subject & (Session & "quality = 0")` | up | `Subject`: 1, 2 | + +**Why.** The seed selects subjects 1 and 2, each with a session of quality 0, and *down* takes all their sessions, not only those. The seed's condition reads `Session`, which the delete empties before `Subject`, so the seed is materialized to the keys of subjects 1 and 2 before the first DELETE. Evaluated live, the DELETE on `Subject` would match nothing, and the subjects would remain ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)). + +### K2. Self-referencing seed + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session & (dj.U("subject_id") & (Session & "quality = 0"))` | down | `Session`: (1, 1), (1, 2), (2, 1), (2, 2)
`Trial`: (1, 1, 1), (1, 2, 1), (2, 1, 1), (2, 2, 1) | +| `Session & (dj.U("subject_id") & (Session & "quality = 0"))` | up | `Session`: (1, 1), (1, 2), (2, 1), (2, 2)
`Subject`: 1, 2 | + +**Why.** The seed selects every session of a subject that has a session of quality 0, and its condition reads `Session` itself, which is deleted no later than the seed. It is materialized before the first DELETE; this also avoids MySQL's refusal of a DELETE whose subquery reads the table being deleted (error 1093). + +### K3. Seed kinds give the same rows + +Each seed below selects sessions (2, 1) and (2, 2), the sessions of subject 2: + +| Kind | Seed | +|---|---| +| dict | `Session & {"subject_id": 2}` | +| list of dicts | `Session & [{"subject_id": 2, "session_id": 1}, {"subject_id": 2, "session_id": 2}]` | +| string | `Session & "subject_id = 2"` | +| string with a subquery | `Session & f"subject_id IN (SELECT subject_id FROM {Subject.full_table_name} WHERE species = 'rat')"` | +| query expression | `Session & (Subject & {"species": "rat"})` | +| `AndList` | `Session & dj.AndList([{"subject_id": 2}, "quality >= 0"])` | +| `Not` | `Session & dj.Not(Flag)`, or `Session - Flag` | + +| Seed | Direction | Rows returned | +|---|---|---| +| any of the above | down | `Session`: (2, 1), (2, 2)
`Trial`: (2, 1, 1), (2, 2, 1) | +| any of the above | up | `Session`: (2, 1), (2, 2)
`Subject`: 2 | +| `Session & dj.Not(Flag)` after inserting `Flag` (3; NULL) | down | `Session`: none
`Trial`: none | +| `Session & dj.Not(Flag)` after inserting `Flag` (3; NULL) | up | `Session`: none
`Subject`: none | + +**Why.** Literal keys cross edges by relabeling and the other kinds cross by restriction; both give the same rows. Upward, only subject 2 is reached, though subject 4 is also a rat, because no selected session references it. The last two rows are the NOT IN NULL trap: `Session & dj.Not(Flag)` compiles to `subject_id NOT IN (SELECT subject_id FROM flag)`, and once that list holds a NULL, SQL's `NOT IN` is never true. The seed then selects no rows, and so does its expansion, in every direction; the materialized form `(Session & dj.Not(Flag)).keys()` is empty too. `expand` takes the rows the database selects for the seed and does not reinterpret its condition. + +### K4. `dj.Top` seed + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session & "quality = 1" & dj.Top(2)` | down | `Session`: (1, 2), (2, 1)
`Trial`: (1, 2, 1), (2, 1, 1) | +| `Session & "quality = 1" & dj.Top(2)` | up | `Session`: (1, 2), (2, 1)
`Subject`: 1, 2 | + +**Why.** Three sessions have quality 1; `dj.Top(2)` keeps the first two in primary-key order. The `Top` is replaced by those two keys before traversal. Ignoring it would delete session (3, 1) as well ([#1589](https://github.com/datajoint/datajoint-python/issues/1589)). + +### K5. Empty seeds + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session & False`, `Session & []`, or `Session & {"subject_id": 4}` | down | `Session`: none
`Trial`: none | +| the same | up | `Session`: none
`Subject`: none | + +**Why.** Each seed selects no rows, so each expansion is empty, and the nodes are still present. Subject 4 exists, but no selected session references it: a key dict crosses upward by restriction, not by relabeling, so `{"subject_id": 4}` does not reach it. + +### K6. Unrestricted seed + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session` | down | `Session`: all five rows
`Trial`: all five rows | +| `Session` | up | `Session`: all five rows
`Subject`: 1, 2, 3 | +| `Session & {"species": "rat"}` | down, up | The same rows as `Session`, and a warning | + +**Why.** An unrestricted seed selects every row. Upward, subject 4 is not reached, because no session references it. `species` is not an attribute of `Session`, so the dict means all rows, as in any restriction. As a seed it raises a warning, because a misspelled attribute would otherwise select every row. + +--- + +## Graph shapes + +### X1. Unloaded downstream schema + +```python +# module lab.py; the only schema this process activates +lab = dj.Schema("shapes_lab") + + +@lab +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +# module analysis.py; declared earlier, never imported by this process +analysis = dj.Schema("shapes_analysis") + + +@analysis +class Score(dj.Manual): + definition = """ + -> Subject + score_id : int32 + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Score` (in `shapes_analysis`) | `subject_id, score_id` | (1, 1), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & {"subject_id": 1}` | down | `Subject`: 1
`Score`: (1, 1) | + +**Why.** `expand` first loads every schema that references a loaded schema through foreign keys, so `shapes_analysis` joins the graph even though none of its classes is in scope. Index its table by its full table name. + +### X2. Group cycle refused at declaration + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Trial(dj.Part): + definition = """ + -> master + trial_id : int32 + """ + + +@schema +class Review(dj.Manual): + definition = """ + -> Session + """ +``` + +Later, a new part is added to `Session` that references `Review`: + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Trial(dj.Part): + definition = """ + -> master + trial_id : int32 + """ + + class Note(dj.Part): + definition = """ + -> master + --- + -> Review + """ +``` + +**Result.** Declaring `Session.Note` raises `DataJointError` before any table is created. The message names the part (`Session.Note`), its parent (`Review`), and the cycle: `Session` → `Review` → `Session.Note`, which belongs to `Session`'s group. No rows change. + +**Why.** A part may reference its master, a sibling part, or a table whose ancestors, with groups taken whole, do not include its master. `Review` depends on `Session`, so `Session.Note` would make the group its own ancestor: a cycle in the group quotient `Q`. That would make every cascade, trace, and `list_tables()` on the connection fail to order the tables (T8). Two groups whose parts reference each other's masters form the same kind of cycle and are refused the same way. + +--- + +## See also + +- [Diagram Specification](diagram.md#traversal-specification) — the definitions these examples follow +- [Cascade Specification](cascade.md) — deletes +- [Upstream Trace Specification](trace.md) — `trace` and `self.upstream` +- [Master-Part Specification](master-part.md) — master-part groups From 096cbebb0906caaca225fca4376643b42573028d Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 16:50:10 -0500 Subject: [PATCH 08/12] docs(specs): rewrite cascade and trace as the down and up cases of expand cascade.md now specifies the delete in terms of D and D1: what each part_integrity mode deletes, the row-level enforce check before any DELETE, the symbolic preview, the delete steps, delete-time materialization and concurrency. This removes the contradiction about whether preview pays for materialization: it does not. trace.md now defines a trace as U(S): upward crossings are always restrictions, a trace includes the parts of ancestor masters and their ancestors, and self.upstream is built from the parents of the row being computed. The published user-facing material is kept. --- src/reference/specs/cascade.md | 231 ++++++++++----------------------- src/reference/specs/trace.md | 110 ++++++++++------ 2 files changed, 138 insertions(+), 203 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index b8f83740..68be32b0 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -1,211 +1,122 @@ # Cascade Specification -This document specifies how DataJoint propagates restrictions across the foreign-key (FK) graph for **cascade delete** and **cascade preview** operations. It defines the propagation rules, the role of `part_integrity`, and the Part-to-Master upward propagation that handles renamed FKs and Part-of-Part chains. +A cascade starts at a restricted **seed** table and deletes, together with the seed's rows, every row that depends on them, in an order the database accepts. It is the `down` case of `Diagram.expand`: the rows it selects are defined once, in the [Diagram specification's Operations](diagram.md#operations), and computed by the rules in [Computation](diagram.md#computation). This page specifies how `Table.delete()` and its preview, `Diagram.cascade()`, use those rows. For the user-facing entry points, see [Delete Data](../../how-to/delete-data.md). For dependent concepts, see [Master-Part](master-part.md), [Diagram](diagram.md), and [Data Manipulation](data-manipulation.md). -Cascade is the **downstream** case of the diagram-traversal algebra — the additive traversal `expand(direction="down")`. The rules it uses (the edge rule R1 and the group rule R2), and why they take the form they do, are derived once in the [Diagram spec's Design Rationale](diagram.md#traversal-algebra); this page specifies the cascade-specific behavior on top of them. +## Entry points -## Overview +- `Table.delete(part_integrity=...)` deletes the rows. +- `Diagram.cascade(table_expr, part_integrity=...)` holds the same rows without deleting them, for `counts()`, iteration, `d[T]`, and display. -A *cascade* starts at a (possibly restricted) **seed** table and propagates the restriction to every table that depends on it via foreign keys, so that a delete or preview affects all dependents consistently. Cascade is invoked by: - -- `Table.delete(part_integrity=...)` — executes the cascade as a delete. -- `Diagram.cascade(table_expr, part_integrity=...).counts()` — preview, no mutation. - -Both follow the same propagation rules; only the terminal step (delete vs. count) differs. +Both compute the same rows from the same plan; only the last step differs. ## Dependency graph -DataJoint loads its FK structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.MultiDiGraph` — a multigraph so that parallel foreign keys, including renamed ones, are each their own edge rather than needing synthetic alias nodes). The graph encodes two kinds of structure: +DataJoint loads its foreign-key structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.MultiDiGraph`, so that parallel foreign keys, including renamed ones, are each their own edge). The graph encodes: | Element | Encodes | |---|---| | **Node** | A table, named by its fully-qualified SQL identifier (e.g. ``` `schema`.`table_name` ```). The graph stores the table's primary key set as node data. | -| **Edge `parent -> child`** | An FK constraint from `child` to `parent`. Edge data: `attr_map` (dict mapping child's FK columns -> parent's referenced columns), `aliased` (true iff any column was renamed), `primary` (true iff the FK is in the child's primary key). | -| **Parallel edges** | A child can reference the same parent through more than one foreign key — for example two renamed (`.proj()`) references — so the same `parent -> child` pair may be connected by multiple edges, each carrying its own `attr_map`/`aliased`/`primary`. | - -The cascade engine operates on a copy of this graph (the `Diagram` class), recording per-table restrictions in `_cascade_restrictions` and the set of restricted attributes in `_restriction_attrs`. +| **Edge `parent -> child`** | A foreign key from `child` to `parent`. Edge data: `attr_map` (dict mapping the child's foreign-key columns to the parent's referenced columns), `aliased` (true if any column was renamed), `primary` (true if the foreign key is in the child's primary key). | +| **Parallel edges** | A child can reference the same parent through more than one foreign key, for example through two renamed (`.proj()`) references. Each is its own edge with its own `attr_map`, `aliased`, and `primary`. | -## Restriction propagation across an edge +`Diagram.cascade()` first calls `dependencies.load_all_downstream()`, which adds every schema that references a loaded schema through foreign keys, including schemas the current process has not activated ([X1](traversal-shapes.md#x1-unloaded-downstream-schema)). -Cascade propagates a restriction across each foreign-key edge by the **edge rule R1** — derived in the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra) — restricting the neighbor by the restricted table, projected/renamed onto the shared foreign-key columns. R1 has three degenerate cases, keyed by whether the foreign key renames columns and whether the restricted attributes are the neighbor's whole primary key: +## What a cascade deletes -- **copy** — non-renamed, and the restricted attributes are the neighbor's primary-key attributes: carry the same predicate unchanged (the columns share names). -- **rename** — renamed foreign key: project the restricted table with its columns renamed through the edge's attribute map to the neighbor's names. -- **project** — non-renamed, but the restricted attributes are *not* (only) the neighbor's primary key: project the restricted table onto the shared foreign-key columns so the join matches on the right columns. +`S` is the set of rows the seed selects. `D(S)` is the closure of `S` under foreign-key children and master-part instances; `D1(S)` is the closure under foreign-key children alone ([definitions](diagram.md#operations)). -Cascade applies R1 **forward** (parent to child) for the delete/preview walk. After each step the child's restricted-attribute set is updated, and the child becomes a propagation source for its own children in the next pass. For `part_integrity="cascade"`, R1 is also applied **upstream** (child to parent) to lift a Part's restriction to its Master — the same rule with tail and head swapped. - -## Cascade flow - -```mermaid -flowchart TB - A[seed table_expr] --> B[Diagram.cascade] - B --> C{seed is a Part?
part_integrity=cascade?} - C -- yes --> D[Upward walk:
_propagate_part_to_master] - C -- no --> E[Forward propagation
over descendants] - D --> E - E --> F{any new master pulled in?} - F -- yes --> E - F -- no --> G[Build subgraph,
count or delete] -``` - -The engine performs **multiple passes** when `part_integrity="cascade"` is in effect: each pass forward-propagates over all `allowed_nodes`, and a pass may pull in a new master (with its descendants) that requires another pass. The loop terminates because the graph is a DAG and only finitely many nodes can be added. - -## Delete-time materialization - -Downward expansion for **delete** carries a constraint that preview (count) does not: a table's restriction must not reference a table that is deleted *earlier* in the plan. - -`Table.delete` executes its per-table deletes in **reverse-topological order** (leaves first — see [Data Manipulation](data-manipulation.md)), so every downstream table is emptied before the table it depends on. If a table's restriction were left as a `QueryExpression` referencing a downstream table — which happens whenever the cascade **seed** is itself restricted by a descendant ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)), or when the master's restriction is left pointing at an already-walked part — that subquery would run against a table already emptied by an earlier step, match zero rows, and silently strand the rows that should have been deleted. - -The fix is to **materialize** any such restriction to a literal key set at plan time, before any row is deleted, while the referenced table still holds its rows. Materialization is applied by delete-order reason on **every backend**, not made backend-conditional: the reverse-topological ordering strands rows regardless of engine. (MySQL additionally rejects a DELETE whose subquery targets the table being modified — error 1093 — but PostgreSQL, which permits that self-reference, still needs materialization for the ordering reason, so this is not a MySQL-only concern.) Where materialization is refused, the engine fails closed with a legible error rather than deleting through a stale subquery. Preview/`counts()` issues no deletes and so pays none of this cost. - -The master-key materialization in [Part-to-Master upward propagation](#materialization-at-the-master) below is the specific instance of this rule for the `part_integrity="cascade"` walk. - -## `part_integrity` modes - -Master-Part integrity reflects the contract that a Master row exists for every Part row that references it. Three modes govern how cascade enforces or relaxes that contract: - -| Mode | Behavior | +| `part_integrity` | Deletes | |---|---| -| `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. The *intent* is row-level (each deleted Part row should have its Master deleted), but the shipped post-check is **table-level** — see [Limitations](#limitations) for the resulting rare false negatives and false positives. | -| `"ignore"` | No upward propagation; no post-check. Master and Part are treated as **regular tables** — each is restricted only through its own foreign keys, and a Part row may be left orphaned when its Master row is preserved. Caller is responsible for the consequences. | -| `"cascade"` | **The master-part group is treated as one item.** Whichever member the cascade encounters first, the whole group comes with it: if the **Master** is reached first, all of its Parts are included; if a **Part** is reached first, the restriction is lifted to its Master (via the upward walk below) and then all of the Master's Parts are included (siblings of the originating Part included). This is the group rule R2 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) applied in either direction of travel. | - -This document focuses on the `"cascade"` mode; the `"enforce"` and `"ignore"` modes do not change the propagation graph. - -## Part-to-Master upward propagation - -When `part_integrity="cascade"` and the cascade reaches (or starts at) a Part node, the engine triggers an **upward walk** of the FK graph from the Part to its Master, applying the edge rule R1 upstream at each edge. Reaching the Master first needs no walk — its Parts are pulled in by ordinary forward propagation. +| `"cascade"` | `D(S)`. Whichever member of a master-part group the cascade reaches, the whole instance goes: a reached part row lifts to its master, and the master brings all its part rows. | +| `"ignore"` | `D1(S)`. Masters and parts are treated as plain tables, so a part row can be deleted while its master and sibling part rows stay. The caller is responsible for the consequences. | +| `"enforce"` (default) | `D1(S)`, if it contains the master of every part row in it. Otherwise `delete()` raises `DataJointError` **before deleting anything**, naming a part row and its master. | -### Identifying the Master +By [T7](diagram.md#derived-results), `enforce` deletes exactly when `D1(S) = D(S)`, that is, when the group rule would add nothing; in that case `"enforce"` and `"cascade"` delete the same rows. -The Master is identified by **naming convention** via `dependencies.extract_master`: a Part table's SQL name ends in `master__partname`, and the Master is the prefix before the `__`. The graph must contain a node with that Master name; otherwise the upward walk is skipped. +!!! version-changed "Changed in 2.4" + - `"enforce"` is checked per part row, before the delete. In 2.3 it was a table-level check after the delete, which could miss an orphaned part row when an unrelated master row was deleted in the same cascade. The 2.4 check is stricter. + - With `"cascade"`, a master joins the delete only if one of its part rows did. In 2.3 the part's condition was copied to the master, which could delete masters that had no matching part row ([#1591](https://github.com/datajoint/datajoint-python/issues/1591); [G1](traversal-shapes.md#g1-part-to-master-lift)). + - A downward copy of a condition across a nullable foreign key could delete rows whose foreign key references nothing, and in a diamond it could miss rows ([#1590](https://github.com/datajoint/datajoint-python/issues/1590); [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column), [S7](traversal-shapes.md#s7-diamond-reached-through-a-secondary-foreign-key)). + - A `dj.Top` seed was ignored ([#1589](https://github.com/datajoint/datajoint-python/issues/1589); [K4](traversal-shapes.md#k4-djtop-seed)). + - A seed restricted by one of its descendants was silently not deleted ([#1496](https://github.com/datajoint/datajoint-python/issues/1496); [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)). -### Walking the FK path +`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. -The walk enumerates **every** simple FK path from Master to Part (`nx.all_simple_edge_paths`), so a Part reachable through more than one foreign-key chain — or through parallel edges between the same pair — is restricted through all of them, combined with OR ([#1492](https://github.com/datajoint/datajoint-python/pull/1492)): +**Parts that violate A1.** The group rule applies to parts whose heading contains the master's primary key under the master's own attribute names ([A1](diagram.md#model-and-assumptions)). A part that holds the master's key only under renamed attributes is traversed as a plain table, with a warning at declaration, in every mode ([G6](traversal-shapes.md#g6-a-part-violating-a1)). -``` -Master -> [intermediate Part(s)] -> Part -``` - -Each edge along a path applies the edge rule R1 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) in the upstream direction, per the edge's metadata (`attr_map`, renamed-or-not). **Intermediate Parts in a Part-of-Part chain are restricted along the way** — not only the Master. - -### Materialization at the Master - -After the upward walk completes, the Master's accumulated restrictions are **materialized** to a literal value tuple via `(master_ft & restrictions).proj().to_arrays()` and stored as a single condition. Subsequent forward propagation from the Master back down to its other Parts then generates `WHERE pk IN (literal-list)` rather than `WHERE pk IN (SELECT ... FROM )`. - -This is the [delete-time materialization](#delete-time-materialization) rule applied here: the originating Part is deleted before the Master (reverse-topological order), so a Master restriction left pointing at that Part would match zero rows and strand the Master — the very compositional-integrity violation the upward walk exists to prevent. Capturing the Master's primary keys while the Part still holds its rows avoids that, and as a bonus avoids the self-referential subquery MySQL rejects with error 1093. - -Intermediate Parts in the chain are **not** materialized — they appear only as restrictions on the path, not as forward-cascade sources, so the self-reference issue doesn't arise there. - -## Seed-is-Part case - -When `Table.delete(part_integrity="cascade")` is invoked on a Part directly (e.g. `Master.PartB.delete(part_integrity="cascade")`), the cascade seed is the Part itself. A leaf Part has no out-edges, so the main propagation loop — which fires the `part_integrity` block from inside the `out_edges` iteration — would not trigger. - -The engine handles this case by invoking the upward walk explicitly for the seed before the main loop runs. After the walk pulls in the Master and any intermediate Parts, the main loop then forward-cascades from the Master to its remaining descendants in the usual way. - -## Worked examples +## Preview -### Example 1: Part-of-Part with renamed FK +`Diagram.cascade(...)` is symbolic. It holds restrictions, not rows, and reads nothing from the database until `counts()`, iteration, or `d[T]` evaluates them. It never materializes keys, with one exception: in a group whose internal foreign key renames the master's key, the master keys are evaluated while the diagram is built ([groups that iterate](diagram.md#groups-that-iterate)). -```python -import datajoint as dj +Its nodes are every table the cascade can reach, including tables with zero matching rows, and `prune()` refuses to remove them: a table empty when the preview is built can gain rows before the delete runs. The counts of a preview equal the counts the delete then removes, unless the data change in between. -schema = dj.Schema("imaging") +## Delete execution -@schema -class Subject(dj.Manual): - definition = """ - subject_id : int32 - """ +`Table.delete()` runs these steps inside one transaction: - class Session(dj.Part): - definition = """ - -> master - session_id : int32 - """ +1. **Plan.** Build the `down` plan for the seed, as for the preview. +2. **Check.** With `"enforce"`, check that `D1(S)` contains the master of each of its part rows. If not, raise before any DELETE. +3. **Materialize.** Replace the restrictions that would read an already-emptied table by the literal keys they select (next section). +4. **Delete.** Delete table by table in reverse topological order: leaves first, the seed's table and other ancestors last. +5. **Backstop.** With `"enforce"`, check at the table level that no part rows were deleted without master rows, and roll back otherwise. - class Recording(dj.Part): - definition = """ - -> Subject.Session.proj(src_subject='subject_id', src_session='session_id') - recording_id : int32 - """ +```mermaid +flowchart TB + A[seed] --> B[plan: D or D1, symbolic] + B --> C{part_integrity = enforce?} + C -- yes --> D{D1 holds every part row's master?} + D -- no --> E[raise, nothing deleted] + D -- yes --> F + C -- no --> F[materialize lifts and non-literal seeds] + F --> G[DELETE in reverse topological order] + G --> H[table-level backstop, then commit] ``` -`Recording`'s columns are `{src_subject, src_session, recording_id}` — none of them are named `subject_id`. The FK from `Subject.Session -> Subject.Recording` is aliased. - -When `(Subject.Recording & {"recording_id": 5}).delete(part_integrity="cascade")` runs: - -1. **Seed-is-Part check.** `extract_master(Recording) == Subject`. Trigger the upward walk. -2. **FK path.** The simple FK path from Master to Part is `[Subject, Subject.Session, Subject.Recording]` (`nx.all_simple_edge_paths`). -3. **Walk reversed** (R1 upstream at each edge). - - Edge `Subject.Session -> Subject.Recording`: renamed FK — the **rename** case. `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`. - - Edge `Subject -> Subject.Session`: not renamed, but the child's restricted attributes `{subject_id, session_id}` are not (only) the parent's primary key `{subject_id}` (`session_id` is not in it) — the **project** case. `Subject` is restricted by `Subject.Session` projected onto its foreign-key columns, which for this primary FK is just `subject_id`. -4. **Materialize Master.** `Subject`'s restriction is fetched into a value tuple; replaces the chained `QueryExpression`. -5. **Forward pass.** Master forward-cascades back down to `Subject.Session` and `Subject.Recording` (and any sibling Parts not on the original path), now with the materialized restriction. - -Without the FK walk (the pre-fix behavior), the engine joined `subject_ft.proj() & recording_ft.proj()` on shared attribute names. `Subject` has `subject_id`; `Recording` has `src_subject`. No shared columns -> empty restriction -> Master not restricted. This is the failure mode from [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1. +With `safemode=True`, `delete()` shows the per-table counts inside the transaction and asks before committing. -### Example 2: Part-of-Part with no Master reference in PartB +## Delete-time materialization -```python -@schema -class Master(dj.Manual): - definition = """ - master_id : int32 - """ +Downward expansion for a delete carries a constraint that the preview does not: a table's restriction must not read a table that is deleted before it, or the table itself. - class PartA(dj.Part): - definition = """ - -> master - part_a_id : int32 - """ +`Table.delete()` deletes in reverse topological order, so every downstream table is emptied before the tables it depends on. A restriction that still read a downstream table when its own DELETE ran would match nothing, and the rows it should delete would stay. This happens when the seed is restricted by one of its descendants ([K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)) or by its own table ([K2](traversal-shapes.md#k2-self-referencing-seed)), and for every part-to-master lift, since a part is deleted before its master. - class PartB(dj.Part): - definition = """ - -> Master.PartA - part_b_id : int32 - """ -``` +Before the first DELETE, the engine therefore materializes each restriction whose *local* terms read a table deleted no later than its own: every lift, and every seed whose restriction is not already literal keys. Terms inherited from a parent are safe, because the parent is deleted after its children. Large key sets are applied in chunks. The full rule is in [Materialization happens only at delete time](diagram.md#materialization-happens-only-at-delete-time). -`PartB`'s definition references `Master.PartA`, not `master` directly. The FK chain Master -> PartA -> PartB still exists in the dependency graph (PartA's FK to Master is `aliased=False`; PartA -> PartB is also `aliased=False`). +The rule follows from the delete order and applies on every backend. MySQL also rejects a DELETE whose subquery reads the table being deleted (error 1093), and PostgreSQL permits it; materializing first avoids both the stranded rows and the error. -For `(Master.PartB & {"master_id": 1}).delete(part_integrity="cascade")`: +## Concurrency -1. Upward walk: PartB -> PartA -> Master (R1 upstream at each edge). - - Edge PartA -> PartB: not renamed, and the child's restricted attributes `{master_id}` are within the parent's primary key `{master_id, part_a_id}` — the **copy** case. `PartA` inherits `PartB`'s restriction directly. - - Edge Master -> PartA: not renamed, and `{master_id}` is the parent's primary key `{master_id}` — the **copy** case. `Master` inherits `PartA`'s restriction. -2. Materialize Master. -3. Forward cascade Master -> PartA -> PartB picks up all sibling rows under `master_id=1`. +Part rows are deleted as `part & (restricted master keys)`, with the master's keys materialized. If another session inserts a part row for one of those masters after the part DELETE, the master's DELETE fails on the part's foreign key (`ON DELETE RESTRICT`), the transaction rolls back, and no part row is left without its master. A foreign-key error during a cascade reports two likely causes: a row inserted concurrently, or a downstream schema the connection cannot see. -Without the FK walk (the pre-fix behavior), the engine jumped directly from PartB to Master via `master_ft.proj() & partB_ft.proj()`. PartA was never restricted, and the chain semantics were silently incorrect. This is [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 2. +## Worked example: a seed restricted by a descendant -## Algorithmic complexity +With the [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant) schema, the seed `Subject & (Session & "quality = 0")` selects subjects 1 and 2. -For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N · E) edge visits per pass, with at most one pass per master pulled in by upward propagation. For typical schemas (small Master-Part groups, shallow chains), the cost is dominated by the materialization fetch at the Master — one `SELECT ... FROM master_ft` over the restricted master rows. +1. **Plan.** `Subject` holds the seed. `Session` holds `Session & Subject_restricted.proj()`, and so on down: `Trial` and `Flag`. +2. **Materialize.** The seed reads `Session`, which is deleted before `Subject`, so it is replaced by the keys of subjects 1 and 2. The other restrictions are inherited from parents and stay symbolic. +3. **Delete.** `Trial`, `Flag`, and `Session` rows of subjects 1 and 2 are deleted first, then `Subject` rows 1 and 2. -## What is not part of this specification +Without step 2, the DELETE on `Subject` would run after the sessions of quality 0 were gone, match nothing, and leave subjects 1 and 2 in place ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)). -- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and applies the same edge rule R1 upstream. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Upstream Trace Specification](trace.md) for `trace`'s API and semantics. -- **Custom propagation rules** (user-defined): not supported. R1 and its three cases (copy, rename, project) cover the cases the FK graph can produce. -- **Cross-schema cascade**: handled by `dependencies.load_all_downstream()` called from `Diagram.cascade()`; orthogonal to the propagation rules described here. +The [Traversal Shape Catalog](traversal-shapes.md) gives the rows for the other shapes: lifts ([G1](traversal-shapes.md#g1-part-to-master-lift)), part-of-part chains ([G2](traversal-shapes.md#g2-part-of-part-with-two-chains)), a second foreign key to the master ([G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master)), chained instances ([G4](traversal-shapes.md#g4-renamed-sibling-foreign-key-chained-across-three-instances)), and merge tables ([G5](traversal-shapes.md#g5-merge-table)). ## Limitations -The following are known, documented behaviors of the cascade engine as shipped: +- **Materialization memory.** Materialized keys are held in memory, one key per selected row of a lifted master or a materialized seed. Large key sets are applied in chunks. +- **Zero-row nodes.** A table the cascade can reach but whose restriction selects nothing appears with a count of 0 in `counts()` and in iteration. This is by design. + +## What is not part of this specification -- **Materialization memory cost.** The master restriction is materialized via `to_arrays()` (required by the reverse-topological delete order — not merely MySQL error 1093; see [Materialization at the Master](#materialization-at-the-master)). The cost is bounded by the number of distinct master rows referenced by the matching parts. Cascade **preview** (`Diagram.cascade(...).counts()`) pays the same materialization cost as an actual delete. -- **Empty-match sentinel.** When no master rows match, the master carries an always-false restriction and appears with zero rows in `counts()` and iteration. This is by design, not an error. -- **Enforce granularity.** The `part_integrity="enforce"` post-check is **table-level**: it verifies that *some* rows of the master table were deleted whenever part rows were, not that each deleted part row's *specific* master row was deleted. As a result, rare false negatives (an unrelated master row happened to be deleted in the same cascade, masking a genuine orphan) and false positives (deleting already-orphaned part rows whose master was removed earlier) are possible. +- **`Diagram.trace()`**: the `up` case of the same traversal; see the [Upstream Trace Specification](trace.md). +- **Custom propagation rules**: not supported. The edge rule and the group rule cover every case the foreign-key graph can produce. ## References -- Source: `src/datajoint/diagram.py` — `Diagram.cascade`, `_propagate_restrictions`, `_apply_propagation_rule`, `_apply_propagation_rule_upward`, `_propagate_part_to_master`. -- Source: `src/datajoint/dependencies.py` — dependency-graph construction, `extract_master`. -- Issue: [datajoint-python #1429](https://github.com/datajoint/datajoint-python/issues/1429) — bug report and motivating examples for the upward propagation rules. -- [Master-Part Specification](master-part.md) — Part-Master contract. -- [Diagram Specification](diagram.md) — graph operations on the dependency graph. -- [Data Manipulation Specification](data-manipulation.md) — `delete()` and `delete_quick()` user-facing API. +- Source: `src/datajoint/diagram.py` (`Diagram.expand`, `Diagram.cascade`), `src/datajoint/dependencies.py` (dependency graph, `topo_sort`, `extract_master`). +- Issues: [#1429](https://github.com/datajoint/datajoint-python/issues/1429) (part-to-master lift with renamed foreign keys and part-of-part chains), [#1496](https://github.com/datajoint/datajoint-python/issues/1496) (delete-time materialization), [#1589](https://github.com/datajoint/datajoint-python/issues/1589), [#1590](https://github.com/datajoint/datajoint-python/issues/1590), [#1591](https://github.com/datajoint/datajoint-python/issues/1591) (rows deleted outside the cascade). +- [Diagram Specification](diagram.md#traversal-specification) — the definitions of `D`, `D1`, and the computation. +- [Traversal Shape Catalog](traversal-shapes.md) — worked examples. +- [Master-Part Specification](master-part.md) — the master-part contract. +- [Data Manipulation Specification](data-manipulation.md) — `delete()` and `delete_quick()`. diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index bbfc1389..2c90440e 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -2,17 +2,15 @@ This document specifies two interlocking features that give DataJoint a first-class **upstream read surface** — a way to query, from any set of rows, the exact ancestor rows they were derived from: -1. **`Diagram.trace(table_expr)`** — the upstream mirror of `Diagram.cascade()`; walks the FK graph from a restricted table expression to every ancestor. +1. **`Diagram.trace(table_expr)`** — the upstream mirror of `Diagram.cascade()`; selects every row the seed's rows depend on. 2. **`self.upstream`** — a property on `AutoPopulate` tables that exposes the trace for the current `key` during `make()` execution. -The two are designed as a unit. `trace` is the underlying graph operation; `self.upstream` is the ergonomic surface for `make()`. Together they make the "read from declared upstream" convention easy to follow and to inspect — which is exactly what makes a `make()` reproducible and statically verifiable (see the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract)). +The two are designed as a unit. `trace` is the underlying graph operation; `self.upstream` is the ergonomic surface for `make()`. Together they make the "read from declared upstream" convention easy to follow and to inspect — which is what makes a `make()` reproducible and statically verifiable (see the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract)). !!! version-added "New in DataJoint 2.3" Introduced together. Existing code that does not use these features is unaffected. -For the related downstream operation, see [Cascade Specification](cascade.md). For `make()` itself, see [AutoPopulate Specification](autopopulate.md). - -Trace is the **upstream** case of the diagram-traversal algebra — the additive traversal `expand(direction="up")`, the mirror of cascade's downstream `expand(direction="down")`. The shared rules (the edge rule R1 and the group rule R2) are derived once in the [Diagram spec's Design Rationale](diagram.md#traversal-algebra); this page specifies the upstream-specific behavior and the `self.upstream` surface on top of them. +A trace is the `up` case of `Diagram.expand`: `Diagram.trace(seed)` equals `Diagram.expand(seed, "up")`. The rows it selects are defined once, in the [Diagram specification's Operations](diagram.md#operations); this page specifies the upstream-specific behavior and the `self.upstream` surface. For the downstream case, see the [Cascade Specification](cascade.md). For `make()` itself, see the [AutoPopulate Specification](autopopulate.md). ## Why this exists @@ -21,26 +19,28 @@ A computed row's reproducibility rests on the convention that `make(self, key)` 1. A way to **construct** the upstream view as a query object: `Diagram.trace()`. 2. A way to **expose** that view ergonomically inside `make()` so users naturally do the right thing: `self.upstream`. -Without (1), downstream tools that need row-level lineage (data-lineage viewers, CDC publishers, audit logs) have to reimplement the FK walk themselves — repeatedly, with subtly different bugs. Without (2), even users who follow the convention write awkward boilerplate (`(Upstream & key).fetch1(...)` everywhere). +Without (1), downstream tools that need row-level lineage (data-lineage viewers, CDC publishers, audit logs) have to reimplement the foreign-key walk themselves — repeatedly, with subtly different bugs. Without (2), even users who follow the convention write awkward boilerplate (`(Upstream & key).fetch1(...)` everywhere). ## Concepts ### Trace as the upstream mirror of cascade -`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"*. They are the two directions of the same traversal (`expand`), over the same dependency graph and the same rules — only the direction differs: +`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"* They are the two directions of one traversal, `expand`, over the same dependency graph and the same rules: -| Method | `expand` direction | Question answered | -|---|---|---| -| `Diagram.cascade(expr)` | `down` | What's affected if these rows are deleted? | -| `Diagram.trace(expr)` | `up` | What contributed to these rows? | +| Method | `expand` direction | Rows | Question answered | +|---|---|---|---| +| `Diagram.cascade(expr, part_integrity="cascade")` | `down` | `D(S)` | What's affected if these rows are deleted? | +| `Diagram.trace(expr)` | `up` | `U(S)` | What contributed to these rows? | -Convergence is always **union** — an ancestor contributes to a child row if it is reachable via *any* foreign-key path. This is reachability, not an intersection; there is no AND-flavored upstream analog. +`U(S)` is the smallest set of rows that contains the seed rows `S`, the rows each of its rows references through a foreign key, and the whole master-part instance of each of its rows. It is consistent on its own: every foreign key of its rows resolves inside it, and every instance in it is whole ([T5](diagram.md#derived-results)). -### Reusing the propagation rules +The two directions are exact mirrors ([T3](diagram.md#derived-results)): a row `y` depends on a row `x` exactly when deleting `x` would delete `y`. Results combine by union: an ancestor contributes if it is reachable through *any* foreign-key path, and there is no intersection. -`trace` applies the **edge rule R1** in the upstream direction — the same rule `cascade` applies downstream — plus the **group rule R2** for master–part. Both are derived once in the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra); a renamed FK is relabelled through the edge's attribute map, Part-of-Part chains are walked naturally, and parallel foreign keys are each handled independently. +### The rules upstream -This is why `trace` cannot ship before the upstream primitives exist in the codebase. As of DataJoint 2.3 they are in place (added with [#1429's cascade fix](https://github.com/datajoint/datajoint-python/pull/1468)), and `trace` is a direct consumer. +`trace` crosses each foreign key upward by **restriction**: the parent is restricted to the rows referenced by at least one restricted child row, `parent & child_restricted.proj(...)`, with renamed attributes mapped back through the foreign key's `attr_map`. A condition on the child is never copied to the parent, because it describes child rows ([R1](diagram.md#r1-the-edge-rule)). A child row whose foreign key has a NULL column references nothing through it. + +The group rule ([R2](diagram.md#r2-the-group-rule)) applies upward as it does downward: a part row brings its master and all of its master's part rows, and a master row brings all its part rows. Parallel foreign keys are each crossed on their own. ### The `make()` read/write boundary @@ -50,11 +50,11 @@ DataJoint's `make(self, key)` convention has always been: Read from self's declared upstream dependencies; write to self (and its Parts). ``` -This is the **read/write boundary**: data flows in from declared ancestors and out to the target table and its Parts. Anything that crosses this boundary in another direction undermines reproducibility — a `fetch` from an undeclared table makes that dependency invisible to the FK graph; an `insert` into another table sneaks rows past it. (The full rule set is the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract).) +This is the **read/write boundary**: data flows in from declared ancestors and out to the target table and its Parts. Anything that crosses this boundary in another direction undermines reproducibility — a `fetch` from an undeclared table makes that dependency invisible to the foreign-key graph; an `insert` into another table sneaks rows past it. (The full rule set is the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract).) `self.upstream` makes the read side of this boundary the path of least resistance: -- **`self.upstream`** is the recommended read channel. Reading through it keeps every dependency visible to the FK graph and pre-restricted to the current `key`. +- **`self.upstream`** is the recommended read channel. Reading through it keeps every dependency visible to the foreign-key graph and pre-restricted to the current `key`. - **`self`** (and its Parts) is the write target. ## 1. `Diagram.trace(table_expr)` @@ -69,28 +69,28 @@ def trace(cls, table_expr: QueryExpression) -> Diagram: | Parameter | Type | Description | |---|---|---| -| `table_expr` | `QueryExpression` | A (possibly restricted) table expression. The trace propagates this restriction upstream through the FK graph. | +| `table_expr` | `QueryExpression` | The seed: one table with any restriction (see [Seeds](diagram.md#seeds)). | -Returns a `Diagram` instance whose nodes are the seed and all of its ancestors (across all loaded schemas), each carrying a restriction derived from the seed. +Returns an expanded `Diagram` in the `up` direction. Its nodes are the seed's table and every table reachable upward from it, across all loaded schemas, including the parts of every master reached. Each node holds that table's rows in `U(S)`, possibly none. ### Behavior -`trace` mirrors `cascade`: +1. Load the dependency graph via `connection.dependencies.load_all_upstream()`, the upstream analog of `load_all_downstream` ([#1423](https://github.com/datajoint/datajoint-python/issues/1423)). It discovers every schema reachable through foreign keys from the seed's schema, including schemas the current process has not activated. +2. Compute `U(S)` in one pass over the tables in reverse topological order: each table is restricted by its seed and by the restricted rows of its children; each master-part group by the masters its members reach, after which each part takes all part rows of those masters ([Computation](diagram.md#computation)). +3. Keep the seed's table and the tables reachable upward, with groups taken whole. Descendants of the seed and unrelated tables are not included. -1. Load the dependency graph via `connection.dependencies.load_all_upstream()` — the upstream analog of `load_all_downstream`, introduced with `Diagram.trace` ([#1423](https://github.com/datajoint/datajoint-python/issues/1423)). It discovers all schemas reachable via reverse FK edges from the seed's schema. -2. Take the seed's restriction and propagate it **upstream** along the FK graph. For each edge `parent -> child`, when the child has a restriction, apply the edge rule R1 upstream (its **copy**, **rename**, or **project** case — see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) to derive the parent's restriction. -3. Trim the resulting graph to **seed + ancestors only**. Descendants of the seed and unrelated ancestors are not included. -4. Convergence is **OR**: an ancestor entity is included if reachable through *any* FK path from the seed (consistent with how a child row "comes from" any of its FK parents). +The trace stays symbolic: no rows are read until a node is indexed, counted, or fetched. The one exception is a group whose internal foreign key renames the master's key, which is evaluated while the trace is built ([groups that iterate](diagram.md#groups-that-iterate)). -The graph traversal is dual to `cascade`'s. `cascade` walks `seed + descendants(seed)`; `trace` walks `seed + ancestors(seed)`. Both apply rules edge-by-edge; both terminate because the dependency graph is a DAG. +!!! version-changed "Changed in 2.4" + A trace includes all parts of every master it reaches, and the ancestors of those parts ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). In 2.3 it included a part only if the part lay on a foreign-key path to the seed. Upward crossings are always restrictions; 2.3 copied some conditions upward, which is correct only for keys of existing child rows. ### Indexing: `trace[T]` -The primary user-facing operation. Returns the ancestor's table pre-restricted through the FK join path. +The primary user-facing operation. Returns the table pre-restricted to its rows in the trace. ```python trace = dj.Diagram.trace(MyChild & key) -session = trace[Session] # QueryExpression: Session restricted to ancestors of MyChild & key +session = trace[Session] # QueryExpression: Session restricted to the rows MyChild & key depends on session.fetch1("session_date") trace[ExtractTraces].to_arrays("trace") ``` @@ -100,16 +100,18 @@ The argument may be: - **A Table subclass** (e.g. `Session`) — convenient when the class is in scope. - **A string** giving a table's fully-qualified or class name (e.g. `"experiment.Session"` or `"Session"`) — for when it isn't. -In both cases the result is the ancestor pre-restricted through the FK join path, returned as a `FreeTable` (a full `QueryExpression`, so every query operator and the fetch API apply). +In both cases the result is a full `QueryExpression`, so every query operator and the fetch API apply. -Requesting a table that is not an ancestor (or the seed itself) of `table_expr` raises `DataJointError`. This is the same guarantee that `cascade` provides for descendants. +Requesting a table that is not a node of the trace raises `DataJointError`. A node with zero rows, such as a part of an ancestor master that has no rows for this seed, returns an empty expression. ### Other methods | Method | Behavior | |---|---| -| `trace.counts()` | `dict[str, int]` mapping each contributing table's full table name (e.g. `` `imaging`.`subject` ``) to its row count under the seed's restriction. Includes the seed alongside its ancestors. Mirror of `cascade.counts()`. | -| `iter(trace)` | Iterate over pre-restricted `FreeTable`s in topological order (parents/roots first). | +| `trace.counts()` | `dict[str, int]` mapping each node's full table name (e.g. `` `imaging`.`subject` ``) to its row count. Includes the seed. Mirror of `cascade.counts()`. | +| `iter(trace)` | Iterate over pre-restricted `FreeTable`s in topological order (parents first). | +| `trace.prune()` | Remove nodes with zero rows. Allowed on traces, unlike cascades (new in 2.4). | +| `trace1 + trace2` | For two traces, the union of their rows, table by table (new in 2.4). | All `QueryExpression` operators (`join`, `aggr`, `proj`, `restrict`, `&`, `*`) work on the values returned by `trace[T]`. The trace is a *view* into the graph; the returned `QueryExpression`s are first-class. @@ -176,28 +178,47 @@ trace.counts() # '`imaging`.`__extract_traces`': 1, '`imaging`.`__summary`': 1} ``` -For a renamed-FK case (paralleling [Cascade Spec §Worked Example 1](cascade.md#example-1-part-of-part-with-renamed-fk)), R1's **rename** case reverses the rename so `trace[Ancestor]` returns the ancestor with its native column names regardless of how the seed's columns are named. +For a renamed foreign key, the restriction maps the attributes back through `attr_map`, so `trace[Ancestor]` returns the ancestor with its own column names however the seed's columns are named ([S4](traversal-shapes.md#s4-renamed-foreign-key)). + +For a **merge table** — a master whose parts each reference a different upstream table — the trace from a downstream row reaches the merge master, its part row, and the upstream table that part names ([G5](traversal-shapes.md#g5-merge-table)): + +```python +trace = dj.Diagram.trace(Analysis & {"recording_id": 1}) +trace[Recording.Ephys] # the part row of recording 1 +trace[EphysSource] # the source that produced recording 1 +``` ## 2. `self.upstream` inside `make()` ### Where it's set -Before invoking `make(self, key)`, the framework records the current key but defers constructing the trace. `self.upstream = Diagram.trace(self & key)` is built the first time the user accesses `self.upstream` inside `make()` and memoized for the rest of the call; `make()` implementations that never read `self.upstream` never build the trace. Both the key and the memoized trace are cleared after `make()` returns (including when it raises). +Before invoking `make(self, key)`, the framework records the current key but defers constructing the trace. The trace is built the first time `make()` accesses `self.upstream`, and memoized for the rest of the call; `make()` implementations that never read `self.upstream` never build it. Both the key and the memoized trace are cleared after `make()` returns (including when it raises). -Once built, the trace diagram is reused for the remainder of the call. The underlying SQL is not issued until the user indexes into an ancestor: `self.upstream[Session]` builds a `QueryExpression`, and `.fetch1()` on that expression triggers a SELECT. Fetch results are not cached — each `self.upstream[T]` access returns a fresh `QueryExpression`, so reading the same ancestor twice issues two SELECTs. +The row for `key` does not exist yet while `make()` runs, so `self.upstream` cannot be a trace of `self & key`, which selects nothing. It is built from the parents that row will have: for each foreign key `e` in the table's primary key, the parent row `P_e & relabel_e(key)` (the key's values renamed onto `P_e`'s primary key), and + +``` +self.upstream = ⋃ over primary foreign keys e of Diagram.trace(P_e & relabel_e(key)) +``` + +the table-by-table union of those traces. Foreign keys outside the primary key are not included, because their values are not known before the insert. See [`self.upstream`](diagram.md#operations) in the Diagram specification. + +!!! version-changed "Changed in 2.4" + In 2.3, `self.upstream` was `Diagram.trace(self & key)`, which returned the parents only because the engine copied the key's condition upward. The 2.4 construction starts from the parent rows directly, and adds the parts of ancestor masters and their ancestors. + +Once built, the trace is reused for the remainder of the call. The underlying SQL is not issued until the user indexes into an ancestor: `self.upstream[Session]` builds a `QueryExpression`, and `.fetch1()` on that expression triggers a SELECT. Fetch results are not cached — each `self.upstream[T]` access returns a fresh `QueryExpression`, so reading the same ancestor twice issues two SELECTs. ### What it returns -`self.upstream[T]` returns the same value `dj.Diagram.trace(self & key)[T]` would return — the ancestor table pre-restricted to the rows that contributed to the current `key`. All `QueryExpression` operators work; the standard fetch API (`fetch`, `fetch1`, `to_arrays`, `to_dicts`, `to_pandas`) is fully supported. +`self.upstream[T]` returns the ancestor table `T` pre-restricted to the rows that the current `key` depends on. All `QueryExpression` operators work; the standard fetch API (`fetch`, `fetch1`, `to_arrays`, `to_dicts`, `to_pandas`) is fully supported. ### Allowed table set `self.upstream` exposes: -- All declared ancestors of `self` (transitively, including renamed-FK chains). -- **All Parts of every ancestor Master.** By the group rule R2 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)), a master-part group is one item: once an ancestor Master is in the trace, its whole part-group comes with it — not only those Parts that happen to lie on an FK path to `self`. This is the upstream mirror of `part_integrity="cascade"` treating the group atomically. +- all ancestors of `self` reached through its primary foreign keys (transitively, including renamed foreign keys); +- **all parts of every ancestor master**, and their ancestors. By the group rule, a master-part group is one entity: once an ancestor master is in the trace, its whole group comes with it, not only those parts that happen to lie on a foreign-key path to `self`. This is the upstream mirror of `part_integrity="cascade"` treating the group as one. -Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. This is the same guarantee `Diagram.trace(...)` provides; `self.upstream` is just the per-`make()` instance of it. +Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. A table's own parts are descendants, not ancestors; read them directly as `self.PartName`. ### Examples @@ -287,6 +308,8 @@ Teams adopt the read surface incrementally: 2. **Start using `self.upstream` in new `make()` implementations.** Reads become ergonomic; nothing else changes. 3. **Migrate existing `make()` implementations** as opportunity allows. Replace `(Upstream & key).fetch(...)` with `self.upstream[Upstream].fetch(...)`. No semantic difference — but the read is now visibly confined to the declared upstream. +In 2.4, traces and `self.upstream` also include the parts of ancestor masters and their ancestors. Code that relied on `trace[T]` raising for such a table now receives its rows. + ## What is not in this specification - **Enforcement of the `make()` contract.** `trace` and `self.upstream` make the contract easy to follow and to inspect, but the open-source framework does not check it at runtime. Confirming that a `make()` reads only from its declared ancestors and writes only to its target, across every access path, is a code-inspection problem — performed by the DataJoint platform's code-deployment CI/CD (static analysis where sound, agentic review for the dynamic cases), backed by commit-pinned governed execution. See the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract). @@ -294,9 +317,10 @@ Teams adopt the read surface incrementally: ## References -- Source (shipped in 2.3): `src/datajoint/diagram.py` (`Diagram.trace`), `src/datajoint/autopopulate.py` (`AutoPopulate.upstream`). Implemented in [#1471](https://github.com/datajoint/datajoint-python/pull/1471) (trace) and [#1473](https://github.com/datajoint/datajoint-python/pull/1473) (self.upstream), building on the cascade rules from [#1468](https://github.com/datajoint/datajoint-python/pull/1468). -- Issues: [#1423](https://github.com/datajoint/datajoint-python/issues/1423) (Diagram.trace), [#1424](https://github.com/datajoint/datajoint-python/issues/1424) (self.upstream). -- [Cascade Specification](cascade.md) — the downstream case of the same edge rule R1 and group rule R2 shared with `trace`. +- Source: `src/datajoint/diagram.py` (`Diagram.expand`, `Diagram.trace`), `src/datajoint/autopopulate.py` (`AutoPopulate.upstream`). First implemented in [#1471](https://github.com/datajoint/datajoint-python/pull/1471) (trace) and [#1473](https://github.com/datajoint/datajoint-python/pull/1473) (self.upstream). +- Issues: [#1423](https://github.com/datajoint/datajoint-python/issues/1423) (Diagram.trace), [#1424](https://github.com/datajoint/datajoint-python/issues/1424) (self.upstream), [#1481](https://github.com/datajoint/datajoint-python/issues/1481) (parts of ancestor masters). +- [Diagram Specification](diagram.md#traversal-specification) — the definitions of `U` and the rules it is computed by. +- [Traversal Shape Catalog](traversal-shapes.md) — worked examples, with the rows each trace returns. +- [Cascade Specification](cascade.md) — the downstream case of the same traversal. - [AutoPopulate Specification](autopopulate.md) — `make()` execution model and the make() reproducibility contract. -- [Diagram Specification](diagram.md) — graph operations on the dependency graph. -- [Entity Integrity](../../explanation/entity-integrity.md) — schema dimensions and FK semantics. +- [Entity Integrity](../../explanation/entity-integrity.md) — schema dimensions and foreign-key semantics. From 40770fe6190a39af122547eb55bb8b485484e3e8 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 17:34:17 -0500 Subject: [PATCH 09/12] docs(specs): define master-part groups by ownership A part belongs to its master's group when a foreign key in its primary key references the master, or an owned part of the same master, with renames allowed. Its owning attributes, the ones that chain maps onto the master's key, define its master row and its instance. A part that is not owned is a plain table in traversal, ordering and the declaration check, and gets a warning. - R2's lift renames the owning attributes onto the master's key. - A group iterates exactly when a foreign key inside it does not map owning attributes onto owning attributes; the one-pass condition is stated the same way. - R1 defines the attributes a condition names (every dict key, the whole heading of a query expression) and notes that 2.3 copies the compiled predicate. - Relabeling materialized keys downward gets an exact condition, with two counterexamples to the looser wording. - T7, T8 and X2 follow the ownership rule of the declaration check. - The catalog's G6 is now a part with a secondary reference to its master, and G7 adds the owned part-of-part chain with renamed attributes. - cascade.md and trace.md follow, state the enforce preview, and relate delete-time materialization to the subquery-seed fix. --- src/reference/specs/cascade.md | 9 ++- src/reference/specs/diagram.md | 56 ++++++++++++------ src/reference/specs/trace.md | 8 +-- src/reference/specs/traversal-shapes.md | 76 ++++++++++++++++++++----- 4 files changed, 111 insertions(+), 38 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 68be32b0..3795bb50 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -29,7 +29,7 @@ DataJoint loads its foreign-key structure into a directed acyclic graph (`Connec | `part_integrity` | Deletes | |---|---| -| `"cascade"` | `D(S)`. Whichever member of a master-part group the cascade reaches, the whole instance goes: a reached part row lifts to its master, and the master brings all its part rows. | +| `"cascade"` | `D(S)`. Whichever member of a master's group the cascade reaches, the whole instance goes: a reached row of an owned part lifts to its master, and the master brings all its owned part rows. | | `"ignore"` | `D1(S)`. Masters and parts are treated as plain tables, so a part row can be deleted while its master and sibling part rows stay. The caller is responsible for the consequences. | | `"enforce"` (default) | `D1(S)`, if it contains the master of every part row in it. Otherwise `delete()` raises `DataJointError` **before deleting anything**, naming a part row and its master. | @@ -39,16 +39,17 @@ By [T7](diagram.md#derived-results), `enforce` deletes exactly when `D1(S) = D(S - `"enforce"` is checked per part row, before the delete. In 2.3 it was a table-level check after the delete, which could miss an orphaned part row when an unrelated master row was deleted in the same cascade. The 2.4 check is stricter. - With `"cascade"`, a master joins the delete only if one of its part rows did. In 2.3 the part's condition was copied to the master, which could delete masters that had no matching part row ([#1591](https://github.com/datajoint/datajoint-python/issues/1591); [G1](traversal-shapes.md#g1-part-to-master-lift)). - A downward copy of a condition across a nullable foreign key could delete rows whose foreign key references nothing, and in a diamond it could miss rows ([#1590](https://github.com/datajoint/datajoint-python/issues/1590); [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column), [S7](traversal-shapes.md#s7-diamond-reached-through-a-secondary-foreign-key)). + - Parts belong to a master's group by ownership rather than by table name ([#1619](https://github.com/datajoint/datajoint-python/issues/1619)). A part-of-part chain with renamed attributes lifts; a part that references its master only through a secondary or nullable foreign key is a plain table. - A `dj.Top` seed was ignored ([#1589](https://github.com/datajoint/datajoint-python/issues/1589); [K4](traversal-shapes.md#k4-djtop-seed)). - A seed restricted by one of its descendants was silently not deleted ([#1496](https://github.com/datajoint/datajoint-python/issues/1496); [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)). `part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. -**Parts that violate A1.** The group rule applies to parts whose heading contains the master's primary key under the master's own attribute names ([A1](diagram.md#model-and-assumptions)). A part that holds the master's key only under renamed attributes is traversed as a plain table, with a warning at declaration, in every mode ([G6](traversal-shapes.md#g6-a-part-violating-a1)). +**Parts the master does not own.** The group rule applies to the parts a master owns: a part whose primary key references the master, directly or through another owned part, possibly under renamed attributes ([A1](diagram.md#model-and-assumptions)). A part that references its master only through a secondary or nullable foreign key, or not at all, is traversed as a plain table in every mode, with a warning at declaration ([G6](traversal-shapes.md#g6-a-part-its-master-does-not-own)). A part-of-part chain with renamed attributes, as in [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1, is owned and lifts to its master ([G7](traversal-shapes.md#g7-owned-part-of-part-with-renamed-attributes)). ## Preview -`Diagram.cascade(...)` is symbolic. It holds restrictions, not rows, and reads nothing from the database until `counts()`, iteration, or `d[T]` evaluates them. It never materializes keys, with one exception: in a group whose internal foreign key renames the master's key, the master keys are evaluated while the diagram is built ([groups that iterate](diagram.md#groups-that-iterate)). +`Diagram.cascade(...)` is symbolic. It holds restrictions, not rows, and reads nothing from the database until `counts()`, iteration, or `d[T]` evaluates them. It never materializes keys, with one exception: in a group where a foreign key between members does not map owning attributes onto owning attributes, the master keys are evaluated while the diagram is built ([groups that iterate](diagram.md#groups-that-iterate)). With `"enforce"`, the preview holds `D1(S)` and does not raise; the check runs when `delete()` starts. Its nodes are every table the cascade can reach, including tables with zero matching rows, and `prune()` refuses to remove them: a table empty when the preview is built can gain rows before the delete runs. The counts of a preview equal the counts the delete then removes, unless the data change in between. @@ -84,6 +85,8 @@ Downward expansion for a delete carries a constraint that the preview does not: Before the first DELETE, the engine therefore materializes each restriction whose *local* terms read a table deleted no later than its own: every lift, and every seed whose restriction is not already literal keys. Terms inherited from a parent are safe, because the parent is deleted after its children. Large key sets are applied in chunks. The full rule is in [Materialization happens only at delete time](diagram.md#materialization-happens-only-at-delete-time). +This generalizes the seed fix in [#1611](https://github.com/datajoint/datajoint-python/pull/1611) (2.3.5), which replaces a delete seed whose conditions contain a subquery by the primary keys it selects, as a `dj.Top` seed already was. In 2.4 the same holds for every seed whose restriction is not literal keys, and for every lift. + The rule follows from the delete order and applies on every backend. MySQL also rejects a DELETE whose subquery reads the table being deleted (error 1093), and PostgreSQL permits it; materializing first avoids both the stranded rows and the error. ## Concurrency diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 41b27f3b..c932a7a3 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -179,7 +179,7 @@ The preview of `(table_expr).delete(part_integrity=...)`. It loads every schema | `"ignore"` | `D1(S)`: the rows reached through foreign keys alone, with part tables treated as plain tables | | `"enforce"` (default) | `D1(S)`. `Table.delete()` deletes these rows only if they include the master of every part row among them, and otherwise raises before deleting anything | -`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. The [Cascade Specification](cascade.md) covers the delete itself. +With `"enforce"`, the preview holds `D1(S)` and does not raise; the check runs in `Table.delete()`. `part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. The [Cascade Specification](cascade.md) covers the delete itself. !!! version-changed "Changed in 2.4" `"enforce"` checks each part row's own master, before any row is deleted. In 2.3 it was a table-level check run after the delete. With `"cascade"`, a master joins the delete only if one of its part rows did ([#1591](https://github.com/datajoint/datajoint-python/issues/1591)). @@ -301,14 +301,27 @@ The definitions are about rows; the computation is about tables and restrictions **Parallel foreign keys.** Two foreign keys between the same pair of tables are separate references, each with its own `attr_map` ([S5](traversal-shapes.md#s5-parallel-foreign-keys)). -**A1, the owning reference.** A part's heading contains its master's primary key under the master's attribute names, with no NULLs, inherited through `-> master` or through a foreign key to a sibling part that satisfies A1 itself. For a part row `r`: +**A1, owned parts.** A part is **owned** by its master if a foreign key in the part's primary key references the master, or references an owned part of the same master. The references along this chain may rename attributes. A foreign key in the primary key cannot be NULL, so every row of an owned part references a row at each step of the chain. A master and its owned parts form the master's **group**. -- `master(r)` is the master row with that key; -- `inst(r)`, the **instance** of `r`, is `master(r)` together with every part row, in any part of the master, that has the same master key. +The **owning attributes** of an owned part are the part's attributes that the chain maps onto the master's primary key: -For a master row `m`, `inst(m)` is `m` with all its part rows. For a row `r` of a plain table, `inst(r) = {r}`. +- for `-> master` in the primary key, the master's own key attributes; +- for `-> master.proj(m_id="master_id")`, the attribute `m_id`; +- for a reference to an owned part, the attributes that the reference maps onto that part's owning attributes ([G7](traversal-shapes.md#g7-owned-part-of-part-with-renamed-attributes)). -A part that violates A1 gets a warning at declaration and is traversed as a plain table: it has no master and no instance ([G6](traversal-shapes.md#g6-a-part-violating-a1)). +If a part has several owning chains, its owning attributes come from an unrenamed `-> master` if it has one, and otherwise from the first owning reference in its definition. + +For a row `r` of an owned part: + +- `master(r)` is the master row whose primary key equals `r`'s owning attributes; +- `inst(r)`, the **instance** of `r`, is `master(r)` together with every row of every owned part of the same master whose owning attributes have the same values. + +For a master row `m`, `inst(m)` is `m` together with the rows of its owned parts whose owning attributes equal `m`'s key. For any other row `r`, `inst(r) = {r}`. + +A part that is not owned gets a warning at declaration, and it is a plain table in traversal, in `topo_sort`, and in the declaration check: it has no master and no instance. This covers a part that does not reference its master, one that references it only through a secondary or nullable foreign key, and one that references it only through a part that is not owned ([G6](traversal-shapes.md#g6-a-part-its-master-does-not-own)). + +!!! version-added "New in 2.4" + Ownership decides which parts belong to a master's group ([#1619](https://github.com/datajoint/datajoint-python/issues/1619)). In 2.3, every part was grouped with its master by its table name. **Row relations.** For a row `r`: @@ -391,13 +404,15 @@ The argument is T4's with `parent` in place of `child`. `U(D(S))` is `U` of `D(S)`, so T5 applies to it with `D(S)` as the seed. For non-idempotence, take a table `A` with two child tables `B` and `C`, and a row `a` with one child `b` in `B` and one child `c` in `C`. With `S = {b}`, `both(S) = U({b}) = {a, b}`, and `both({a, b}) = U({a, b, c}) = {a, b, c}`: the second application picks up `a`'s other descendant. -**T7, enforce.** Under A1, `D1(S)` is lift-closed if and only if `D1(S) = D(S)`. +**T7, enforce.** `D1(S)` is lift-closed if and only if `D1(S) = D(S)`. + +Every row of an owned part can be reached from its master row by `child` steps along its owning chain: each reference in the chain lies in a primary key, so it is never NULL, and it maps owning attributes onto owning attributes, so it stays in the same instance. So if `D1(S)` contains the master of each of its part rows, it contains their whole instances, is closed under `down`, and equals `D(S)`. Conversely, `D(S)` is closed under `inst`, so if `D1(S) = D(S)` it is lift-closed. `enforce` therefore deletes only when the group rule would add nothing. -Under A1, every part row of an instance can be reached from its master row by `child` steps, through `-> master` or through sibling parts that carry the same key. So if `D1(S)` contains the master of each of its part rows, it contains their whole instances, is closed under `down`, and equals `D(S)`. Conversely, `D(S)` is closed under `inst`, so if `D1(S) = D(S)` it is lift-closed. `enforce` therefore deletes only when the group rule would add nothing. +**T8, ordering.** `topo_sort` succeeds if and only if the group quotient `Q` is acyclic. The **group quotient** `Q` is the table-level foreign-key graph with each master and its owned parts merged into a single node; a part that is not owned stays a node of its own. -**T8, ordering.** Under A1, `topo_sort` succeeds if and only if the group quotient `Q` is acyclic. The **group quotient** `Q` is the table-level foreign-key graph with each master and its parts merged into a single node. +`topo_sort` keeps each owned part right after its master, so it sorts the table graph after adding, for each owned part, an edge from each of the part's parents outside its group to its master. The master reaches each owned part along its owning chain, so a cycle in `Q` becomes a cycle in that graph. Conversely, a cycle in that graph cannot stay inside one group, because foreign keys between tables never form a cycle, so it maps to a cycle in `Q`. -`topo_sort` sorts the table graph after adding, for each part, an edge from each of the part's parents outside its group to its master. Under A1 the master reaches each of its parts, so a cycle in `Q` becomes a cycle in that graph. Conversely, a cycle in that graph cannot stay inside one group, because foreign keys between tables never form a cycle, so it maps to a cycle in `Q`. Declaring a part that would close a cycle in `Q` is refused ([X2](traversal-shapes.md#x2-group-cycle-refused-at-declaration)), so `Q` stays acyclic. +The declaration check keeps `Q` acyclic ([#1619](https://github.com/datajoint/datajoint-python/issues/1619); [X2](traversal-shapes.md#x2-group-cycle-refused-at-declaration)). An owned part may reference its master, an owned part of the same master, or a table that does not depend on its master, with groups taken whole; parts the master does not own count as outside the group. A reference to any other table would close a cycle in `Q`, and the declaration is refused before any table is created. ### Computation @@ -411,13 +426,16 @@ The engine computes the rows above as restrictions on tables, without reading th C & P_restricted.proj() ``` -The parent's condition may instead be **copied** to the child unchanged, but only when the attributes it names are among `e`'s attributes, `e` is not renamed, and none of `e`'s columns is nullable. Then each child row holds the same values under the same names as the parent row it references, and it references one, so the copied condition selects exactly the children. [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column) shows the nullable case, [S4](traversal-shapes.md#s4-renamed-foreign-key) the renamed case, and [S7](traversal-shapes.md#s7-diamond-reached-through-a-secondary-foreign-key) a condition on an attribute the child holds through a different parent. +The parent's condition may instead be **copied** to the child unchanged, but only when the attributes it names are among `e`'s attributes, `e` is not renamed, and none of `e`'s columns is nullable. Here, the attributes a condition names are every key of a dict, including keys that are not attributes of the parent, and the whole heading of a query expression. Under these conditions each child row holds the same values under the same names as the parent row it references, and it references one, so the copied condition selects exactly the children. [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column) shows the nullable case, [S4](traversal-shapes.md#s4-renamed-foreign-key) the renamed case, and [S7](traversal-shapes.md#s7-diamond-reached-through-a-secondary-foreign-key) a condition on an attribute the child holds through a different parent. -A copy is a shortcut: it must select the same rows as the restriction, so the condition must also mean the same thing on the child as on the parent. Two kinds of condition can change meaning when moved, and are not copied: +A copy is a shortcut, and it must select the same rows as the restriction. Two kinds of condition would change meaning if moved, and the rule above already keeps them from being copied: - a dict with keys that are not attributes of the parent: they are ignored on the parent but can match attributes of the child; - a query expression that shares more attributes with the child than with the parent: a restriction by a query expression matches on all shared attributes. Copied to `Session`, the seed condition of [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant), `Session & "quality = 0"`, would match on every attribute of `Session` and select only the sessions of quality 0. Copied the same way, the seed `Subject - (Session & "quality = 0")` would select every session of another quality, including sessions of subjects that are not deleted. +!!! note + The 2.3 engine copies the compiled SQL predicate, which reads only the parent's columns. That is why 2.3 copies dict conditions, and conditions given as query expressions, correctly. The 2.4 plan refers to conditions rather than to compiled SQL, so the rule above applies. + **Parallel foreign keys** combine with OR: a child row is reached if any of its foreign keys references a restricted row. **Up.** Crossing `e` from a restricted child `C` to the parent `P` is always the restriction @@ -434,7 +452,7 @@ The one shortcut upward applies to keys materialized from the child table itself For a master `M` and its part `Pi`, with `R(T)` the restriction on table `T`: -- **lift:** `M & R(Pi).proj(*M.primary_key)` — the masters of the restricted part rows. A1 puts the master's key in every part row, so the lift is one step on that key, with no walk along the foreign keys between parts ([G2](traversal-shapes.md#g2-part-of-part-with-two-chains)). Another foreign key from the part to its master is not used for the lift ([G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master)). +- **lift:** `M & R(Pi).proj(m1="o1", m2="o2", …)` — the masters of the restricted part rows. Each owning attribute `o` of `Pi` is renamed to the attribute `m` of the master's key that it maps onto; for an unrenamed `-> master` this is `M & R(Pi).proj()`. Every owned part row holds its owning attributes, so the lift is one step, with no walk along the foreign keys between parts ([G2](traversal-shapes.md#g2-part-of-part-with-two-chains), [G7](traversal-shapes.md#g7-owned-part-of-part-with-renamed-attributes)). The part's other references to its master are not used for the lift ([G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master)). - **re-expand:** `Pi & R(M)` — every part row of the restricted masters. The lift is existential: a master joins only if at least one of its part rows did ([G1](traversal-shapes.md#g1-part-to-master-lift)). @@ -446,18 +464,20 @@ Because `Q` is acyclic (T8), the engine visits each node of `Q` once: in `topo_s - A **plain table** `T` gets `R(T)` = its seed ∪ R1 from each restricted neighbor on the incoming side (its parents for `down`, its children for `up`). - A **group** with master `M` gets `R(M)` = the seed on `M` ∪ R1 into `M` ∪ lift(R1 into each part ∪ the seed on each part). Then each part `Pi` gets `Pi & R(M)`. -`up` is the same pass with every edge reversed. The pass is exact because every neighbor on the incoming side is final before it is used (`Q` is acyclic), and because a foreign key inside a group maps the master's key to itself, so steps inside a group never leave the instances already in `R(M)`. The next subsection covers the groups where that fails. +`up` is the same pass with every edge reversed. The pass is exact because every neighbor on the incoming side is final before it is used (`Q` is acyclic), and because every foreign key inside a group maps owning attributes onto owning attributes, so steps inside a group never leave the instances already in `R(M)`. The next subsection covers the groups where that fails. #### Groups that iterate -A foreign key inside a group that renames the master's key links different instances of the group. In [G4](traversal-shapes.md#g4-renamed-sibling-foreign-key-chained-across-three-instances), a `Carry` row of run 2 references a `Step` row of run 1; in [G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master), a part references another row of its own master. The engine detects such a foreign key from its `attr_map`. For that group only, it evaluates the lifted master keys and repeats the group step until no new key appears. This is the only place where a traversal reads rows before a delete. +A group iterates if and only if some foreign key inside it does not map the referencing part's owning attributes onto the referenced table's owning attributes (for the master, its primary key). Such a foreign key links different instances of the group. In [G4](traversal-shapes.md#g4-renamed-sibling-foreign-key-chained-across-three-instances), a `Carry` row of run 2 references a `Step` row of run 1 through `prev_run`, which is not an owning attribute of `Carry`; in [G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master), a part references another row of its own master through `other_session`. Renaming the master's key is one way this happens. A foreign key references the whole primary key, which contains the referenced table's owning attributes, so every reference inside a group carries a master key; the question is only whether it is the referencing part's own. + +The engine detects such a foreign key from its `attr_map`. For that group only, it evaluates the lifted master keys and repeats the group step until no new key appears. This is the only place where a traversal reads rows before a delete. #### Seeds - A seed is one table with any restriction. Projections, joins, aggregations, and unions are refused as seeds. The restriction itself may use them, for example a subquery on a descendant ([K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)) or on the seed's own table ([K2](traversal-shapes.md#k2-self-referencing-seed)). - A `dj.Top` seed is first replaced by the primary keys it selects ([K4](traversal-shapes.md#k4-djtop-seed)). - A dict that names none of the table's attributes means all rows, as in any restriction ([Query Algebra §2.4](query-algebra.md#24-dictionary-conditions)). As a seed, it raises a warning ([K6](traversal-shapes.md#k6-unrestricted-seed)). -- A materialized seed (a dict, a list of dicts, or `.keys()`) may cross an edge by **relabeling**: keeping only the keys the foreign key carries, renamed through `attr_map`. Relabeling is used where it selects the same rows as R1: downward, when the foreign key has no nullable column or the keys name all its columns; upward, only for keys read from the child table that contain all its columns. A live seed — a string or a query expression — crosses by restriction. Both give the same rows ([K3](traversal-shapes.md#k3-seed-kinds-give-the-same-rows)). +- A materialized seed (a dict, a list of dicts, or `.keys()`) may cross an edge by **relabeling**: keeping only the keys the foreign key carries, renamed through `attr_map`. Relabeling is used only where it selects the same rows as R1. Downward, every key must name, among the parent's attributes, only attributes the foreign key carries; and if the foreign key has a nullable column, every key must name all of the foreign key's attributes, none of them NULL. Otherwise relabeling changes the rows: `Session & {"quality": 0}` relabels to `{}` across `Trial`'s foreign key and would select every trial, and on the schema of [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column), `Surgery & {"subject_id": 1, "surgery_id": None}` selects no surgery but, relabeled, would select session (1, 2; NULL). Upward, relabeling applies only to keys read from the child table that contain all of the foreign key's columns. A live seed — a string or a query expression — crosses by restriction. Both give the same rows ([K3](traversal-shapes.md#k3-seed-kinds-give-the-same-rows)). - The attributes a string condition names are read from its text outside any subquery. #### The plan @@ -487,11 +507,11 @@ Without this, a restriction that reads a table emptied earlier in the same delet A diagram returned by `expand()`, `cascade()`, or `trace()` is an **expanded diagram**. It has a direction, and it obeys the following contract. -- **Nodes.** The node set is the *table-level reach* of the seed's table: the tables reached by following foreign keys in the direction of travel, together with the master and all parts of every group that has a reached member. (For `D1`, the group step is left out.) For `both`, it is the upward reach of the downward reach. A node may hold zero rows. +- **Nodes.** The node set is the *table-level reach* of the seed's table: the tables reached by following foreign keys in the direction of travel, together with the master and all owned parts of every group that has a reached member. (For `D1`, the group step is left out.) For `both`, it is the upward reach of the downward reach. A node may hold zero rows. - **`d[T]`** returns `T` restricted to its rows, in every direction, and raises `DataJointError` if `T` is not a node. - **`counts()`** works, and **iteration** runs in topological order. - **Copying** a diagram, `dj.Diagram(d)`, keeps its direction. -- **`d1 + d2`** of two diagrams with the same direction holds, for each table, the union of their rows (T2). Adding diagrams of different directions raises. +- **`d1 + d2`** of two diagrams with the same direction holds, for each table, the union of their rows (T2). It raises for diagrams of different directions, for an expanded diagram and one that is not expanded, and for two `cascade()` diagrams with different `part_integrity` modes. - **`-`, `*`, `+ n`, and `- n`** raise. - **`prune()`** raises for `down` and is allowed for `up` and `both`. diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 2c90440e..3bb6b939 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -40,7 +40,7 @@ The two directions are exact mirrors ([T3](diagram.md#derived-results)): a row ` `trace` crosses each foreign key upward by **restriction**: the parent is restricted to the rows referenced by at least one restricted child row, `parent & child_restricted.proj(...)`, with renamed attributes mapped back through the foreign key's `attr_map`. A condition on the child is never copied to the parent, because it describes child rows ([R1](diagram.md#r1-the-edge-rule)). A child row whose foreign key has a NULL column references nothing through it. -The group rule ([R2](diagram.md#r2-the-group-rule)) applies upward as it does downward: a part row brings its master and all of its master's part rows, and a master row brings all its part rows. Parallel foreign keys are each crossed on their own. +The group rule ([R2](diagram.md#r2-the-group-rule)) applies upward as it does downward: a row of an owned part brings its master and the master's owned part rows, and a master row brings its owned part rows. A part that its master does not own is a plain table ([A1](diagram.md#model-and-assumptions)). Parallel foreign keys are each crossed on their own. ### The `make()` read/write boundary @@ -79,10 +79,10 @@ Returns an expanded `Diagram` in the `up` direction. Its nodes are the seed's ta 2. Compute `U(S)` in one pass over the tables in reverse topological order: each table is restricted by its seed and by the restricted rows of its children; each master-part group by the masters its members reach, after which each part takes all part rows of those masters ([Computation](diagram.md#computation)). 3. Keep the seed's table and the tables reachable upward, with groups taken whole. Descendants of the seed and unrelated tables are not included. -The trace stays symbolic: no rows are read until a node is indexed, counted, or fetched. The one exception is a group whose internal foreign key renames the master's key, which is evaluated while the trace is built ([groups that iterate](diagram.md#groups-that-iterate)). +The trace stays symbolic: no rows are read until a node is indexed, counted, or fetched. The one exception is a group where a foreign key between members does not map owning attributes onto owning attributes; its master keys are evaluated while the trace is built ([groups that iterate](diagram.md#groups-that-iterate)). !!! version-changed "Changed in 2.4" - A trace includes all parts of every master it reaches, and the ancestors of those parts ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). In 2.3 it included a part only if the part lay on a foreign-key path to the seed. Upward crossings are always restrictions; 2.3 copied some conditions upward, which is correct only for keys of existing child rows. + A trace includes all owned parts of every master it reaches, and the ancestors of those parts ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). In 2.3 it included a part only if the part lay on a foreign-key path to the seed. Upward crossings are always restrictions; 2.3 copied some conditions upward, which is correct only for keys of existing child rows. ### Indexing: `trace[T]` @@ -216,7 +216,7 @@ Once built, the trace is reused for the remainder of the call. The underlying SQ `self.upstream` exposes: - all ancestors of `self` reached through its primary foreign keys (transitively, including renamed foreign keys); -- **all parts of every ancestor master**, and their ancestors. By the group rule, a master-part group is one entity: once an ancestor master is in the trace, its whole group comes with it, not only those parts that happen to lie on a foreign-key path to `self`. This is the upstream mirror of `part_integrity="cascade"` treating the group as one. +- **all owned parts of every ancestor master**, and their ancestors. By the group rule, a master-part group is one entity: once an ancestor master is in the trace, its whole group comes with it, not only those parts that happen to lie on a foreign-key path to `self`. This is the upstream mirror of `part_integrity="cascade"` treating the group as one. Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. A table's own parts are descendants, not ancestors; read them directly as `self.PartName`. diff --git a/src/reference/specs/traversal-shapes.md b/src/reference/specs/traversal-shapes.md index e5660e00..71144be9 100644 --- a/src/reference/specs/traversal-shapes.md +++ b/src/reference/specs/traversal-shapes.md @@ -28,7 +28,8 @@ Each shape has a tiny schema, a few rows, one or more seeds, and the exact rows | [G3](#g3-a-second-renamed-foreign-key-to-its-own-master) | A second, renamed foreign key to its own master | Only the owning reference lifts | | [G4](#g4-renamed-sibling-foreign-key-chained-across-three-instances) | Renamed sibling foreign key, chained across three instances | The group step iterates | | [G5](#g5-merge-table) | Merge table | A trace reaches the parts of an ancestor master | -| [G6](#g6-a-part-violating-a1) | A part violating A1 | Traversed as a plain table | +| [G6](#g6-a-part-its-master-does-not-own) | A part its master does not own | Traversed as a plain table | +| [G7](#g7-owned-part-of-part-with-renamed-attributes) | Owned part-of-part with renamed attributes | The lift renames the owning attributes | | [K1](#k1-seed-restricted-by-a-descendant) | Seed restricted by a descendant | Seeds are materialized before deleting | | [K2](#k2-self-referencing-seed) | Self-referencing seed | Same, for a seed that reads its own table | | [K3](#k3-seed-kinds-give-the-same-rows) | Seed kinds give the same rows | Dict, list, string, subquery, `AndList`, `Not` | @@ -425,7 +426,7 @@ class Session(dj.Manual): """ ``` -`Session.Link` belongs to its session through `-> master` (the owning reference, A1) and points to another session through a renamed foreign key. +`Session.Link` is owned through `-> master`, so its owning attribute is `session_id`. It points to another session through a second, renamed foreign key. | Table | Columns | Rows | |---|---|---| @@ -437,7 +438,7 @@ class Session(dj.Manual): | `Session.Link & {"session_id": 1}` | down | `Session.Link`: (1, 1), (3, 1)
`Session`: 1, 3 | | `Session.Link & {"session_id": 1}` | up | `Session.Link`: (1, 1)
`Session`: 1, 2 | -**Why.** Link (1, 1) lifts through its owning reference to session 1, not to session 2. Deleting session 1 then deletes link (3, 1), which points to it, and so the rest of instance 3. Upward, link (1, 1) reads session 2, a parent, but not link (3, 1), a child. The renamed foreign key links one instance to another, so this group is [iterated](diagram.md#groups-that-iterate). +**Why.** Link (1, 1) lifts through its owning attribute to session 1, not to session 2. Deleting session 1 then deletes link (3, 1), which points to it, and so the rest of instance 3. Upward, link (1, 1) reads session 2, a parent, but not link (3, 1), a child. The second foreign key maps `other_session`, which is not an owning attribute, onto the master's key; it links one instance to another, so this group [iterates](diagram.md#groups-that-iterate). With `part_integrity="enforce"`, `(Session & {"session_id": 1}).delete()` raises: it would delete link (3, 1) without session 3. @@ -480,7 +481,7 @@ A `Carry` row of one run references a `Step` row of an earlier run. | `Run & {"run_id": 2}` | up | `Run`: 1, 2
`Run.Step`: (1, 1), (2, 1)
`Run.Carry`: (2, 1) | | `Run & {"run_id": 2}` | both | `Run`: 1, 2, 3
`Run.Step`: (1, 1), (2, 1), (3, 1)
`Run.Carry`: (2, 1), (3, 1) | -**Why.** Deleting run 1 deletes step (1, 1), so carry (2, 1), which references it, and so the rest of run 2; that reaches carry (3, 1) and run 3. Run 4 is untouched. One pass over the group would stop at run 1. The engine detects the renamed key and repeats the group step: the master keys grow as {1}, {1, 2}, {1, 2, 3}, and a fourth round adds nothing. Upward the chain runs the other way, from run 2 back to run 1. +**Why.** Deleting run 1 deletes step (1, 1), so carry (2, 1), which references it, and so the rest of run 2; that reaches carry (3, 1) and run 3. Run 4 is untouched. One pass over the group would stop at run 1. `Carry`'s reference to `Step` maps `prev_run`, not `Carry`'s owning attribute `run_id`, onto `Step`'s owning attribute, so the engine repeats the group step: the master keys grow as {1}, {1, 2}, {1, 2, 3}, and a fourth round adds nothing. Upward the chain runs the other way, from run 2 back to run 1. ### G5. Merge table @@ -551,7 +552,7 @@ trace[ImagingSource] # empty: a node with zero rows, not an error **Why.** Upward, recording 1 brings its whole instance. Its `Ephys` part row references the source that produced the recording, so `trace[EphysSource]` resolves ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). Downward, deleting ephys source 2 deletes the part row that names it, so the whole of recording 2, and its analysis. With `part_integrity="enforce"` that delete raises, because the part row would go without its master. -### G6. A part violating A1 +### G6. A part its master does not own ```python @schema @@ -562,25 +563,74 @@ class Probe(dj.Manual): class Channel(dj.Part): definition = """ - -> master.proj(host_id="probe_id") channel_id : int32 + --- + -> master """ ``` -`Probe.Channel` references its master only under a renamed key, so its heading does not contain `probe_id` and it violates A1. Declaring it raises a warning, and it is traversed as a plain table: no lift and no instance. +`Probe.Channel` references its master only through a secondary foreign key, so the master does not own it. Declaring it raises a warning, and it is traversed as a plain table: no lift and no instance. A nullable reference, `-> [nullable] master`, has the same effect. | Table | Columns | Rows | |---|---|---| | `Probe` | `probe_id` | 1, 2 | -| `Probe.Channel` | `host_id, channel_id` | (1, 1), (1, 2), (2, 1) | +| `Probe.Channel` | `channel_id; probe_id` | (1; 1), (2; 1), (3; 2) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Probe.Channel & {"channel_id": 1}` | down | `Probe.Channel`: 1 | +| `Probe.Channel & {"channel_id": 1}` | up | `Probe.Channel`: 1
`Probe`: 1 | +| `Probe & {"probe_id": 1}` | down | `Probe`: 1
`Probe.Channel`: 1, 2 | + +**Why.** As a plain table, the channel reaches its probe only as a parent: downward nothing lifts, and upward channel 2 is not part of channel 1's instance. Deleting the probe still deletes its channels, as children. Moving the reference into the primary key, as `-> master` or renamed as `-> master.proj(host_id="probe_id")`, would make the part owned. + +### G7. Owned part-of-part with renamed attributes + +The part-of-part shape of [#1429](https://github.com/datajoint/datajoint-python/issues/1429), Case 1: + +```python +@schema +class Master(dj.Manual): + definition = """ + master_id : int32 + """ + + class PartA(dj.Part): + definition = """ + -> master + part_a_id : int32 + """ + + class PartB(dj.Part): + definition = """ + -> Master.PartA.proj(src_master="master_id", src_part="part_a_id") + part_b_id : int32 + """ + + +@schema +class Report(dj.Manual): + definition = """ + -> Master + report_id : int32 + """ +``` + +`PartB` has no attribute named `master_id`. It is owned through its primary-key reference to `PartA`, which is owned, and its owning attribute is `src_master`, which that reference maps onto `PartA`'s `master_id`. + +| Table | Columns | Rows | +|---|---|---| +| `Master` | `master_id` | 1, 2, 3 | +| `Master.PartA` | `master_id, part_a_id` | (1, 10), (1, 11), (2, 20) | +| `Master.PartB` | `src_master, src_part, part_b_id` | (1, 10, 100), (1, 11, 110), (2, 20, 200) | +| `Report` | `master_id, report_id` | (1, 1), (2, 1) | | Seed | Direction | Rows returned | |---|---|---| -| `Probe.Channel & {"host_id": 1, "channel_id": 1}` | down | `Probe.Channel`: (1, 1) | -| `Probe.Channel & {"host_id": 1, "channel_id": 1}` | up | `Probe.Channel`: (1, 1)
`Probe`: 1 | -| `Probe & {"probe_id": 1}` | down | `Probe`: 1
`Probe.Channel`: (1, 1), (1, 2) | +| `Master.PartB & {"part_b_id": 100}` | down | `Master.PartB`: (1, 10, 100), (1, 11, 110)
`Master`: 1
`Master.PartA`: (1, 10), (1, 11)
`Report`: (1, 1) | +| `Master.PartB & {"part_b_id": 100}` | up | `Master.PartB`: (1, 10, 100), (1, 11, 110)
`Master`: 1
`Master.PartA`: (1, 10), (1, 11) | -**Why.** As a plain table, the channel reaches its probe only as a parent: downward nothing lifts, and upward channel (1, 2) is not part of channel (1, 1)'s instance. Deleting the probe still deletes its channels, as children. +**Why.** The lift renames the owning attribute onto the master's key, `Master & R(PartB).proj(master_id="src_master")`, and selects master 1 in one step. Master 1 brings its whole instance, including part B row (1, 11, 110), which the seed did not name, and, downward, its report. The reference from `PartB` to `PartA` maps `src_master` onto `PartA`'s owning attribute, so the group needs no iteration. With `part_integrity="enforce"`, deleting the seed raises, because part B row (1, 10, 100) would go without master 1. --- @@ -792,7 +842,7 @@ class Session(dj.Manual): **Result.** Declaring `Session.Note` raises `DataJointError` before any table is created. The message names the part (`Session.Note`), its parent (`Review`), and the cycle: `Session` → `Review` → `Session.Note`, which belongs to `Session`'s group. No rows change. -**Why.** A part may reference its master, a sibling part, or a table whose ancestors, with groups taken whole, do not include its master. `Review` depends on `Session`, so `Session.Note` would make the group its own ancestor: a cycle in the group quotient `Q`. That would make every cascade, trace, and `list_tables()` on the connection fail to order the tables (T8). Two groups whose parts reference each other's masters form the same kind of cycle and are refused the same way. +**Why.** `Session.Note` is owned, because `-> master` is in its primary key. An owned part may reference its master, an owned part of the same master, or a table that does not depend on its master, with groups taken whole; parts the master does not own count as outside the group ([#1619](https://github.com/datajoint/datajoint-python/issues/1619)). `Review` depends on `Session`, so `Session.Note` would make the group its own ancestor: a cycle in the group quotient `Q`, and DataJoint could no longer order these tables for delete, drop, or trace (T8). Two groups whose owned parts reference each other's masters form the same kind of cycle and are refused the same way. A part that is not owned is a plain table, so its references cannot close a group cycle. --- From 0b612d66a09de84a504305fa4d9972f73b361e45 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 17:35:25 -0500 Subject: [PATCH 10/12] docs(specs): note the 2.3 seed failure fixed in 2.3.5 --- src/reference/specs/cascade.md | 2 +- src/reference/specs/diagram.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 3795bb50..68ce356f 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -85,7 +85,7 @@ Downward expansion for a delete carries a constraint that the preview does not: Before the first DELETE, the engine therefore materializes each restriction whose *local* terms read a table deleted no later than its own: every lift, and every seed whose restriction is not already literal keys. Terms inherited from a parent are safe, because the parent is deleted after its children. Large key sets are applied in chunks. The full rule is in [Materialization happens only at delete time](diagram.md#materialization-happens-only-at-delete-time). -This generalizes the seed fix in [#1611](https://github.com/datajoint/datajoint-python/pull/1611) (2.3.5), which replaces a delete seed whose conditions contain a subquery by the primary keys it selects, as a `dj.Top` seed already was. In 2.4 the same holds for every seed whose restriction is not literal keys, and for every lift. +This generalizes the 2.3.5 seed fix ([#1610](https://github.com/datajoint/datajoint-python/issues/1610)), which replaces a delete seed whose conditions contain a subquery by the primary keys it selects, as a `dj.Top` seed already was. In 2.4 the same holds for every seed whose restriction is not literal keys, and for every lift. The rule follows from the delete order and applies on every backend. MySQL also rejects a DELETE whose subquery reads the table being deleted (error 1093), and PostgreSQL permits it; materializing first avoids both the stranded rows and the error. diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index c932a7a3..02798858 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -434,7 +434,7 @@ A copy is a shortcut, and it must select the same rows as the restriction. Two k - a query expression that shares more attributes with the child than with the parent: a restriction by a query expression matches on all shared attributes. Copied to `Session`, the seed condition of [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant), `Session & "quality = 0"`, would match on every attribute of `Session` and select only the sessions of quality 0. Copied the same way, the seed `Subject - (Session & "quality = 0")` would select every session of another quality, including sessions of subjects that are not deleted. !!! note - The 2.3 engine copies the compiled SQL predicate, which reads only the parent's columns. That is why 2.3 copies dict conditions, and conditions given as query expressions, correctly. The 2.4 plan refers to conditions rather than to compiled SQL, so the rule above applies. + The 2.3 engine copies the compiled SQL predicate, which reads only the parent's columns. That is why 2.3 copies dict conditions, and conditions given as query expressions, correctly. (A seed restricted by a query expression failed in 2.3 for a different reason, before any copy; 2.3.5 replaces such a seed by its keys, [#1610](https://github.com/datajoint/datajoint-python/issues/1610).) The 2.4 plan refers to conditions rather than to compiled SQL, so the rule above applies. **Parallel foreign keys** combine with OR: a child row is reached if any of its foreign keys references a restricted row. From 1e9faf4b10bf84ba41441437e5cc257c33fdfdbe Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 17:47:28 -0500 Subject: [PATCH 11/12] docs(specs): define a primary-key reference by its attributes --- src/reference/specs/diagram.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 02798858..e42881bf 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -301,7 +301,7 @@ The definitions are about rows; the computation is about tables and restrictions **Parallel foreign keys.** Two foreign keys between the same pair of tables are separate references, each with its own `attr_map` ([S5](traversal-shapes.md#s5-parallel-foreign-keys)). -**A1, owned parts.** A part is **owned** by its master if a foreign key in the part's primary key references the master, or references an owned part of the same master. The references along this chain may rename attributes. A foreign key in the primary key cannot be NULL, so every row of an owned part references a row at each step of the chain. A master and its owned parts form the master's **group**. +**A1, owned parts.** A part is **owned** by its master if a foreign key in the part's primary key references the master, or references an owned part of the same master. A foreign key is in the primary key when all its attributes are primary-key attributes of the part, wherever the reference is declared. The references along this chain may rename attributes. A foreign key in the primary key cannot be NULL, so every row of an owned part references a row at each step of the chain. A master and its owned parts form the master's **group**. The **owning attributes** of an owned part are the part's attributes that the chain maps onto the master's primary key: @@ -318,7 +318,7 @@ For a row `r` of an owned part: For a master row `m`, `inst(m)` is `m` together with the rows of its owned parts whose owning attributes equal `m`'s key. For any other row `r`, `inst(r) = {r}`. -A part that is not owned gets a warning at declaration, and it is a plain table in traversal, in `topo_sort`, and in the declaration check: it has no master and no instance. This covers a part that does not reference its master, one that references it only through a secondary or nullable foreign key, and one that references it only through a part that is not owned ([G6](traversal-shapes.md#g6-a-part-its-master-does-not-own)). +A part that is not owned gets a warning at declaration, and it is a plain table in traversal, in `topo_sort`, and in the declaration check: it has no master and no instance. This covers a part that does not reference its master, one that references it only through a foreign key with attributes outside the primary key (a secondary or nullable reference), and one that references it only through a part that is not owned ([G6](traversal-shapes.md#g6-a-part-its-master-does-not-own)). !!! version-added "New in 2.4" Ownership decides which parts belong to a master's group ([#1619](https://github.com/datajoint/datajoint-python/issues/1619)). In 2.3, every part was grouped with its master by its table name. From 937b546758b1015af58ef588dada47dda7d22054 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 19:33:22 -0500 Subject: [PATCH 12/12] docs(specs): settle the planner's wording questions --- src/reference/specs/diagram.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index e42881bf..4f5c0e7b 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -309,7 +309,7 @@ The **owning attributes** of an owned part are the part's attributes that the ch - for `-> master.proj(m_id="master_id")`, the attribute `m_id`; - for a reference to an owned part, the attributes that the reference maps onto that part's owning attributes ([G7](traversal-shapes.md#g7-owned-part-of-part-with-renamed-attributes)). -If a part has several owning chains, its owning attributes come from an unrenamed `-> master` if it has one, and otherwise from the first owning reference in its definition. +If a part has several owning chains, its owning attributes come from an unrenamed `-> master` if it has one, and otherwise from the owning reference whose attributes come first in the part's heading, which follows the order of the definition. For a row `r` of an owned part: @@ -416,7 +416,7 @@ The declaration check keeps `Q` acyclic ([#1619](https://github.com/datajoint/da ### Computation -The engine computes the rows above as restrictions on tables, without reading the rows, except where noted. +The engine computes the rows above as restrictions on tables, without reading the rows, except where noted. In the formulas below, a projection `.proj(...)` keeps only the attributes it lists, renamed as shown, so no other attribute of the source takes part in the match. #### R1, the edge rule @@ -482,11 +482,11 @@ The engine detects such a foreign key from its `attr_map`. For that group only, #### The plan -The engine first builds a **plan**: an intermediate form, not SQL. Its nodes are `Seed`, `Restrict(src, attr_map)`, `Lift(part)`, `Reexpand(master)`, `Or`, and `Literal`. A node refers to the restrictions it uses by a handle to another node; compiled SQL is never copied from one restriction into another. The plan compiles to query expressions for the database. It can also be evaluated against rows held in memory, which is how the test suite compares it with the definitions above. +The engine first builds a **plan**: an intermediate form, not SQL. Its nodes are `Seed`, `Restrict(src, attr_map)`, `Lift(part)`, `Reexpand(master)`, `Or`, and `Literal`. A node refers to the restrictions it uses by a handle to another node; compiled SQL is not copied from one restriction into another, except that a copy under R1 applies the seed's own compiled condition to the child. The plan compiles to query expressions for the database. It can also be evaluated against rows held in memory, which is how the test suite compares it with the definitions above. #### Materialization happens only at delete time -Traversal and preview (`counts()`, iteration, `d[T]`) stay symbolic, apart from [groups that iterate](#groups-that-iterate). Before the first DELETE, the engine **materializes** a node, replacing its restriction by the literal keys it selects, when one of the node's local terms reads a table that is deleted no later than the node itself. A *local term* is the node's seed or a lift into it. The terms a node inherits through R1 from a parent are safe: `Table.delete()` deletes in reverse topological order, so a parent still holds its rows when its children are deleted. +Traversal and preview (`counts()`, iteration, `d[T]`) stay symbolic, apart from [groups that iterate](#groups-that-iterate) and `dj.Top` seeds, which are replaced by their keys when the traversal starts. Before the first DELETE, the engine **materializes** a node, replacing its restriction by the literal keys it selects, when one of the node's local terms reads a table that is deleted no later than the node itself. A *local term* is the node's seed or a lift into it. The terms a node inherits through R1 from a parent are safe: `Table.delete()` deletes in reverse topological order, so a parent still holds its rows when its children are deleted. In practice: