diff --git a/mkdocs.yaml b/mkdocs.yaml index 29b5c7d8..624f9925 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -119,6 +119,7 @@ nav: - Primary Keys: reference/specs/primary-keys.md - Fetch API: reference/specs/fetch-api.md - Diagram: reference/specs/diagram.md + - Traversal Shapes: reference/specs/traversal-shapes.md - Type System: - Types: reference/specs/type-system.md - Codec API: reference/specs/codec-api.md diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 8832567c..68ce356f 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -1,216 +1,125 @@ # Cascade Specification -This document specifies how DataJoint propagates restrictions across the foreign-key (FK) graph for **cascade delete** and **cascade preview** operations. It defines the propagation rules, the role of `part_integrity`, and the Part-to-Master upward propagation that handles renamed FKs and Part-of-Part chains. +A cascade starts at a restricted **seed** table and deletes, together with the seed's rows, every row that depends on them, in an order the database accepts. It is the `down` case of `Diagram.expand`: the rows it selects are defined once, in the [Diagram specification's Operations](diagram.md#operations), and computed by the rules in [Computation](diagram.md#computation). This page specifies how `Table.delete()` and its preview, `Diagram.cascade()`, use those rows. For the user-facing entry points, see [Delete Data](../../how-to/delete-data.md). For dependent concepts, see [Master-Part](master-part.md), [Diagram](diagram.md), and [Data Manipulation](data-manipulation.md). -## Overview +## Entry points -A *cascade* starts at a (possibly restricted) **seed** table and propagates the restriction to every table that depends on it via foreign keys, so that a delete or preview affects all dependents consistently. Cascade is invoked by: +- `Table.delete(part_integrity=...)` deletes the rows. +- `Diagram.cascade(table_expr, part_integrity=...)` holds the same rows without deleting them, for `counts()`, iteration, `d[T]`, and display. -- `Table.delete(part_integrity=...)` — executes the cascade as a delete. -- `Diagram.cascade(table_expr, part_integrity=...).counts()` — preview, no mutation. - -Both follow the same propagation rules; only the terminal step (delete vs. count) differs. +Both compute the same rows from the same plan; only the last step differs. ## Dependency graph -DataJoint loads its FK structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.DiGraph`). The graph encodes two kinds of structure: +DataJoint loads its foreign-key structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.MultiDiGraph`, so that parallel foreign keys, including renamed ones, are each their own edge). The graph encodes: | Element | Encodes | |---|---| | **Node** | A table, named by its fully-qualified SQL identifier (e.g. ``` `schema`.`table_name` ```). The graph stores the table's primary key set as node data. | -| **Edge `parent → child`** | An FK constraint from `child` to `parent`. Edge data: `attr_map` (dict mapping child's FK columns → parent's referenced columns), `aliased` (true iff any column was renamed), `primary` (true iff the FK is in the child's primary key). | -| **Parallel edges** | A child can reference the same parent through more than one foreign key — for example two renamed (`.proj()`) references — so the same `parent → child` pair may be connected by multiple edges, each carrying its own `attr_map`/`aliased`/`primary`. | - -The cascade engine operates on a copy of this graph (the `Diagram` class), recording per-table restrictions in `_cascade_restrictions` and the set of restricted attributes in `_restriction_attrs`. - -## Restriction propagation rules - -When propagating a restriction across an edge `parent → child`, one of three rules applies. The rule depends on whether the FK renames columns (`aliased`) and on whether the parent's currently-restricted attributes (`parent_attrs`) are contained in the child's primary key (`child_pk`). - -### Forward propagation (parent → child) +| **Edge `parent -> child`** | A foreign key from `child` to `parent`. Edge data: `attr_map` (dict mapping the child's foreign-key columns to the parent's referenced columns), `aliased` (true if any column was renamed), `primary` (true if the foreign key is in the child's primary key). | +| **Parallel edges** | A child can reference the same parent through more than one foreign key, for example through two renamed (`.proj()`) references. Each is its own edge with its own `attr_map`, `aliased`, and `primary`. | -| Rule | Trigger | Effect on child | -|---|---|---| -| **F1. Copy** | `not aliased and parent_attrs` **non-empty** `and parent_attrs ⊆ child_pk` | Child inherits the parent's restriction directly (same attribute names; literal restriction values copy as-is). An empty attribute set takes rule 3. | -| **F2. Aliased rename** | `aliased` | Child's restriction is `parent.proj(**{fk_col: parent_col for fk_col, parent_col in attr_map.items()})` — the parent expression with columns renamed to match the child's column names. | -| **F3. Project** | `not aliased and parent_attrs ⊄ child_pk` | Child's restriction is `parent.proj()` — the parent projected to its primary key. | +`Diagram.cascade()` first calls `dependencies.load_all_downstream()`, which adds every schema that references a loaded schema through foreign keys, including schemas the current process has not activated ([X1](traversal-shapes.md#x1-unloaded-downstream-schema)). -After applying the rule, the child's restricted-attributes set is updated to track what's now constrained on it. The child becomes a propagation source for its own children in the next pass. +## What a cascade deletes -### Upward propagation (child → parent) +`S` is the set of rows the seed selects. `D(S)` is the closure of `S` under foreign-key children and master-part instances; `D1(S)` is the closure under foreign-key children alone ([definitions](diagram.md#operations)). -Symmetric inverses of the forward rules. Used by `part_integrity="cascade"` to propagate a Part's restriction up to its Master through the FK chain. Edge metadata is the same; the direction of travel is reversed. - -| Rule | Trigger | Effect on parent | -|---|---|---| -| **U1. Copy** | `not aliased and child_attrs` **non-empty** `and child_attrs ⊆ parent_pk` | Parent inherits the child's restriction directly (shared attribute names). An empty attribute set takes rule 3. | -| **U2. Aliased reverse-rename** | `aliased` | Parent's restriction is `child.proj(**{parent_col: fk_col for fk_col, parent_col in attr_map.items()})` — the child expression with FK columns renamed back to the parent's column names. | -| **U3. Project** | `not aliased and child_attrs ⊄ parent_pk` | Parent's restriction is `child.proj(*attr_map.keys())` — the child projected onto its FK columns (which, when non-aliased, share names with the parent's PK) so the parent restriction joins on the right columns. | - -## Cascade flow - -```mermaid -flowchart TB - A[seed table_expr] --> B[Diagram.cascade] - B --> C{seed is a Part?
part_integrity=cascade?} - C -- yes --> D[Upward walk:
_propagate_part_to_master] - C -- no --> E[Forward propagation
over descendants] - D --> E - E --> F{any new master pulled in?} - F -- yes --> E - F -- no --> G[Build subgraph,
count or delete] -``` - -The engine performs **multiple passes** when `part_integrity="cascade"` is in effect: each pass forward-propagates over all `allowed_nodes`, and a pass may pull in a new master (with its descendants) that requires another pass. The loop terminates because the graph is a DAG and only finitely many nodes can be added. - -## `part_integrity` modes - -Master-Part integrity reflects the contract that a Master row exists for every Part row that references it. Three modes govern how cascade enforces or relaxes that contract: - -| Mode | Behavior | +| `part_integrity` | Deletes | |---|---| -| `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. The *intent* is row-level (each deleted Part row should have its Master deleted), but the shipped post-check is **table-level** — see [Limitations](#limitations) for the resulting rare false negatives and false positives. | -| `"ignore"` | No upward propagation; no post-check. Use when the Master row is intentionally preserved and the user has accepted that Part rows may be orphaned. Caller is responsible for the consequences. | -| `"cascade"` | **Upward propagation enabled.** When cascade reaches a Part, the Master is also restricted (via the upward rules below), and the Master then forward-cascades back down to **all** its Parts (siblings of the originating Part included). Used when the user wants the master-part group treated atomically. | - -This document focuses on the `"cascade"` mode; the `"enforce"` and `"ignore"` modes do not change the propagation graph. - -## Part-to-Master upward propagation - -When `part_integrity="cascade"` and the cascade reaches (or starts at) a Part node, the engine triggers an **upward walk** of the FK graph from the Part to its Master, applying the upward rules (U1, U2, U3) at each edge. - -### Identifying the Master +| `"cascade"` | `D(S)`. Whichever member of a master's group the cascade reaches, the whole instance goes: a reached row of an owned part lifts to its master, and the master brings all its owned part rows. | +| `"ignore"` | `D1(S)`. Masters and parts are treated as plain tables, so a part row can be deleted while its master and sibling part rows stay. The caller is responsible for the consequences. | +| `"enforce"` (default) | `D1(S)`, if it contains the master of every part row in it. Otherwise `delete()` raises `DataJointError` **before deleting anything**, naming a part row and its master. | -The Master is identified by **naming convention** via `dependencies.extract_master`: a Part table's SQL name ends in `master__partname`, and the Master is the prefix before the `__`. The graph must contain a node with that Master name; otherwise the upward walk is skipped. +By [T7](diagram.md#derived-results), `enforce` deletes exactly when `D1(S) = D(S)`, that is, when the group rule would add nothing; in that case `"enforce"` and `"cascade"` delete the same rows. -### Walking the FK path +!!! version-changed "Changed in 2.4" + - `"enforce"` is checked per part row, before the delete. In 2.3 it was a table-level check after the delete, which could miss an orphaned part row when an unrelated master row was deleted in the same cascade. The 2.4 check is stricter. + - With `"cascade"`, a master joins the delete only if one of its part rows did. In 2.3 the part's condition was copied to the master, which could delete masters that had no matching part row ([#1591](https://github.com/datajoint/datajoint-python/issues/1591); [G1](traversal-shapes.md#g1-part-to-master-lift)). + - A downward copy of a condition across a nullable foreign key could delete rows whose foreign key references nothing, and in a diamond it could miss rows ([#1590](https://github.com/datajoint/datajoint-python/issues/1590); [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column), [S7](traversal-shapes.md#s7-diamond-reached-through-a-secondary-foreign-key)). + - Parts belong to a master's group by ownership rather than by table name ([#1619](https://github.com/datajoint/datajoint-python/issues/1619)). A part-of-part chain with renamed attributes lifts; a part that references its master only through a secondary or nullable foreign key is a plain table. + - A `dj.Top` seed was ignored ([#1589](https://github.com/datajoint/datajoint-python/issues/1589); [K4](traversal-shapes.md#k4-djtop-seed)). + - A seed restricted by one of its descendants was silently not deleted ([#1496](https://github.com/datajoint/datajoint-python/issues/1496); [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)). -The walk uses `nx.shortest_path(master, part)` to find the FK chain from Master to Part: +`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. -``` -Master → [intermediate Part(s)] → Part -``` - -Each edge along the path fires one upward rule (U1, U2, or U3) per the edge's metadata (`attr_map`, `aliased`). **Intermediate Parts in a Part-of-Part chain are restricted along the way** — not only the Master. - -### Materialization at the Master - -After the upward walk completes, the Master's accumulated restrictions are **materialized** to a literal value tuple via `(master_ft & restrictions).proj().to_arrays()` and stored as a single condition. Subsequent forward propagation from the Master back down to its other Parts then generates `WHERE pk IN (literal-list)` rather than `WHERE pk IN (SELECT ... FROM )`. - -**Why materialization matters.** This is required for correctness on **every backend**, not merely to satisfy MySQL. `Table.delete` executes per-table deletes in reverse-topological order (leaves first — see [Data Manipulation](data-manipulation.md)), so the originating Part is deleted *before* the Master. If the Master's restriction were left as a `QueryExpression` referencing that Part, the Master's own DELETE — issued last — would find the Part already emptied, match zero rows, and silently strand the Master (the very compositional-integrity violation the upward walk exists to prevent). Materializing the Master's primary keys to a literal value set at plan time, before any rows are deleted, captures them while the Part still exists. - -A secondary consequence: the literal set also avoids a self-referential subquery. Left as a query, the Master forward-cascading back to the originating Part would generate a DELETE whose subquery targets the table being modified — which MySQL rejects ("error 1093: You can't specify target table 'T' for update in FROM clause"). PostgreSQL permits that self-reference, but the reverse-topological ordering above means materialization is required on **both** backends regardless — so this must not be treated as a MySQL-only concern. - -Intermediate Parts in the chain are **not** materialized — they appear only as restrictions on the path, not as forward-cascade sources, so the self-reference issue doesn't arise there. - -## Seed-is-Part case - -When `Table.delete(part_integrity="cascade")` is invoked on a Part directly (e.g. `Master.PartB.delete(part_integrity="cascade")`), the cascade seed is the Part itself. A leaf Part has no out-edges, so the main propagation loop — which fires the `part_integrity` block from inside the `out_edges` iteration — would not trigger. +**Parts the master does not own.** The group rule applies to the parts a master owns: a part whose primary key references the master, directly or through another owned part, possibly under renamed attributes ([A1](diagram.md#model-and-assumptions)). A part that references its master only through a secondary or nullable foreign key, or not at all, is traversed as a plain table in every mode, with a warning at declaration ([G6](traversal-shapes.md#g6-a-part-its-master-does-not-own)). A part-of-part chain with renamed attributes, as in [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1, is owned and lifts to its master ([G7](traversal-shapes.md#g7-owned-part-of-part-with-renamed-attributes)). -The engine handles this case by invoking the upward walk explicitly for the seed before the main loop runs. After the walk pulls in the Master and any intermediate Parts, the main loop then forward-cascades from the Master to its remaining descendants in the usual way. +## Preview -## Worked examples +`Diagram.cascade(...)` is symbolic. It holds restrictions, not rows, and reads nothing from the database until `counts()`, iteration, or `d[T]` evaluates them. It never materializes keys, with one exception: in a group where a foreign key between members does not map owning attributes onto owning attributes, the master keys are evaluated while the diagram is built ([groups that iterate](diagram.md#groups-that-iterate)). With `"enforce"`, the preview holds `D1(S)` and does not raise; the check runs when `delete()` starts. -### Example 1: Part-of-Part with renamed FK +Its nodes are every table the cascade can reach, including tables with zero matching rows, and `prune()` refuses to remove them: a table empty when the preview is built can gain rows before the delete runs. The counts of a preview equal the counts the delete then removes, unless the data change in between. -```python -import datajoint as dj +## Delete execution -schema = dj.Schema("imaging") +`Table.delete()` runs these steps inside one transaction: -@schema -class Subject(dj.Manual): - definition = """ - subject_id : int32 - """ +1. **Plan.** Build the `down` plan for the seed, as for the preview. +2. **Check.** With `"enforce"`, check that `D1(S)` contains the master of each of its part rows. If not, raise before any DELETE. +3. **Materialize.** Replace the restrictions that would read an already-emptied table by the literal keys they select (next section). +4. **Delete.** Delete table by table in reverse topological order: leaves first, the seed's table and other ancestors last. +5. **Backstop.** With `"enforce"`, check at the table level that no part rows were deleted without master rows, and roll back otherwise. - class Session(dj.Part): - definition = """ - -> master - session_id : int32 - """ - - class Recording(dj.Part): - definition = """ - -> Subject.Session.proj(src_subject='subject_id', src_session='session_id') - recording_id : int32 - """ +```mermaid +flowchart TB + A[seed] --> B[plan: D or D1, symbolic] + B --> C{part_integrity = enforce?} + C -- yes --> D{D1 holds every part row's master?} + D -- no --> E[raise, nothing deleted] + D -- yes --> F + C -- no --> F[materialize lifts and non-literal seeds] + F --> G[DELETE in reverse topological order] + G --> H[table-level backstop, then commit] ``` -`Recording`'s columns are `{src_subject, src_session, recording_id}` — none of them are named `subject_id`. The FK from `Subject.Session → Subject.Recording` is aliased. +With `safemode=True`, `delete()` shows the per-table counts inside the transaction and asks before committing. -When `(Subject.Recording & {"recording_id": 5}).delete(part_integrity="cascade")` runs: +## Delete-time materialization -1. **Seed-is-Part check.** `extract_master(Recording) == Subject`. Trigger the upward walk. -2. **FK path.** `shortest_path(Subject, Recording) = [Subject, Subject.Session, Subject.Recording]`. -3. **Walk reversed.** - - Edge `Subject.Session → Subject.Recording`: `aliased=True`. Apply **U2** — `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`. - - Edge `Subject → Subject.Session`: `aliased=False`, `child_attrs={subject_id, session_id} ⊆ parent_pk={subject_id}`? No (`session_id` not in parent pk). Apply **U3** — `Subject` is restricted by `Subject.Session.proj(*attr_map.keys())`, projecting the child onto its FK columns; for this primary FK those columns are just `subject_id`, so this is equivalent to `Subject.Session.proj()` projected to `subject_id`. -4. **Materialize Master.** `Subject`'s restriction is fetched into a value tuple; replaces the chained `QueryExpression`. -5. **Forward pass.** Master forward-cascades back down to `Subject.Session` and `Subject.Recording` (and any sibling Parts not on the original path), now with the materialized restriction. +Downward expansion for a delete carries a constraint that the preview does not: a table's restriction must not read a table that is deleted before it, or the table itself. -Without the FK walk (the pre-fix behavior), the engine joined `subject_ft.proj() & recording_ft.proj()` on shared attribute names. `Subject` has `subject_id`; `Recording` has `src_subject`. No shared columns → empty restriction → Master not restricted. This is the failure mode from [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1. +`Table.delete()` deletes in reverse topological order, so every downstream table is emptied before the tables it depends on. A restriction that still read a downstream table when its own DELETE ran would match nothing, and the rows it should delete would stay. This happens when the seed is restricted by one of its descendants ([K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)) or by its own table ([K2](traversal-shapes.md#k2-self-referencing-seed)), and for every part-to-master lift, since a part is deleted before its master. -### Example 2: Part-of-Part with no Master reference in PartB +Before the first DELETE, the engine therefore materializes each restriction whose *local* terms read a table deleted no later than its own: every lift, and every seed whose restriction is not already literal keys. Terms inherited from a parent are safe, because the parent is deleted after its children. Large key sets are applied in chunks. The full rule is in [Materialization happens only at delete time](diagram.md#materialization-happens-only-at-delete-time). -```python -@schema -class Master(dj.Manual): - definition = """ - master_id : int32 - """ +This generalizes the 2.3.5 seed fix ([#1610](https://github.com/datajoint/datajoint-python/issues/1610)), which replaces a delete seed whose conditions contain a subquery by the primary keys it selects, as a `dj.Top` seed already was. In 2.4 the same holds for every seed whose restriction is not literal keys, and for every lift. - class PartA(dj.Part): - definition = """ - -> master - part_a_id : int32 - """ +The rule follows from the delete order and applies on every backend. MySQL also rejects a DELETE whose subquery reads the table being deleted (error 1093), and PostgreSQL permits it; materializing first avoids both the stranded rows and the error. - class PartB(dj.Part): - definition = """ - -> Master.PartA - part_b_id : int32 - """ -``` +## Concurrency -`PartB`'s definition references `Master.PartA`, not `master` directly. The FK chain Master → PartA → PartB still exists in the dependency graph (PartA's FK to Master is `aliased=False`; PartA → PartB is also `aliased=False`). +Part rows are deleted as `part & (restricted master keys)`, with the master's keys materialized. If another session inserts a part row for one of those masters after the part DELETE, the master's DELETE fails on the part's foreign key (`ON DELETE RESTRICT`), the transaction rolls back, and no part row is left without its master. A foreign-key error during a cascade reports two likely causes: a row inserted concurrently, or a downstream schema the connection cannot see. -For `(Master.PartB & {"master_id": 1}).delete(part_integrity="cascade")`: +## Worked example: a seed restricted by a descendant -1. Upward walk: PartB → PartA → Master. - - Edge PartA → PartB: `aliased=False`, `child_attrs ⊆ parent_pk`? `child_attrs = {master_id}`, `parent_pk = {master_id, part_a_id}`. Yes ⊆. Apply **U1** — `PartA` inherits `PartB`'s restriction directly. - - Edge Master → PartA: `aliased=False`, `child_attrs = {master_id} ⊆ parent_pk = {master_id}`. Apply **U1** — `Master` inherits `PartA`'s restriction. -2. Materialize Master. -3. Forward cascade Master → PartA → PartB picks up all sibling rows under `master_id=1`. +With the [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant) schema, the seed `Subject & (Session & "quality = 0")` selects subjects 1 and 2. -Without the FK walk (the pre-fix behavior), the engine jumped directly from PartB to Master via `master_ft.proj() & partB_ft.proj()`. PartA was never restricted, and the chain semantics were silently incorrect. This is [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 2. +1. **Plan.** `Subject` holds the seed. `Session` holds `Session & Subject_restricted.proj()`, and so on down: `Trial` and `Flag`. +2. **Materialize.** The seed reads `Session`, which is deleted before `Subject`, so it is replaced by the keys of subjects 1 and 2. The other restrictions are inherited from parents and stay symbolic. +3. **Delete.** `Trial`, `Flag`, and `Session` rows of subjects 1 and 2 are deleted first, then `Subject` rows 1 and 2. -## Algorithmic complexity +Without step 2, the DELETE on `Subject` would run after the sessions of quality 0 were gone, match nothing, and leave subjects 1 and 2 in place ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)). -For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N · E) edge visits per pass, with at most one pass per master pulled in by upward propagation. For typical schemas (small Master-Part groups, shallow chains), the cost is dominated by the materialization fetch at the Master — one `SELECT ... FROM master_ft` over the restricted master rows. - -## What is not part of this specification - -- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and reuses the same upward rules (U1/U2/U3) defined above. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Upstream Trace Specification](trace.md) for `trace`'s API and semantics. -- **Custom propagation rules** (user-defined): not supported. The three forward and three upward rules cover the cases the FK graph can produce. -- **Cross-schema cascade**: handled by `dependencies.load_all_downstream()` called from `Diagram.cascade()`; orthogonal to the propagation rules described here. +The [Traversal Shape Catalog](traversal-shapes.md) gives the rows for the other shapes: lifts ([G1](traversal-shapes.md#g1-part-to-master-lift)), part-of-part chains ([G2](traversal-shapes.md#g2-part-of-part-with-two-chains)), a second foreign key to the master ([G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master)), chained instances ([G4](traversal-shapes.md#g4-renamed-sibling-foreign-key-chained-across-three-instances)), and merge tables ([G5](traversal-shapes.md#g5-merge-table)). ## Limitations -The following are known, documented behaviors of the cascade engine as shipped: +- **Materialization memory.** Materialized keys are held in memory, one key per selected row of a lifted master or a materialized seed. Large key sets are applied in chunks. +- **Zero-row nodes.** A table the cascade can reach but whose restriction selects nothing appears with a count of 0 in `counts()` and in iteration. This is by design. + +## What is not part of this specification -- **Single FK path (part→master walk).** The upward walk uses `nx.shortest_path` to find the FK chain from a Part to its Master. If a Part reaches its Master through multiple distinct FK chains, restrictions carried by the non-shortest paths are not applied. Workaround: use `part_integrity="ignore"` and perform the additional deletes manually. -- **Materialization memory cost.** The master restriction is materialized via `to_arrays()` (required by the reverse-topological delete order — not merely MySQL error 1093; see [Materialization at the Master](#materialization-at-the-master)). The cost is bounded by the number of distinct master rows referenced by the matching parts. Cascade **preview** (`Diagram.cascade(...).counts()`) pays the same materialization cost as an actual delete. -- **Empty-match sentinel.** When no master rows match, the master carries an always-false restriction and appears with zero rows in `counts()` and iteration. This is by design, not an error. -- **Enforce granularity.** The `part_integrity="enforce"` post-check is **table-level**: it verifies that *some* rows of the master table were deleted whenever part rows were, not that each deleted part row's *specific* master row was deleted. As a result, rare false negatives (an unrelated master row happened to be deleted in the same cascade, masking a genuine orphan) and false positives (deleting already-orphaned part rows whose master was removed earlier) are possible. +- **`Diagram.trace()`**: the `up` case of the same traversal; see the [Upstream Trace Specification](trace.md). +- **Custom propagation rules**: not supported. The edge rule and the group rule cover every case the foreign-key graph can produce. ## References -- Source: `src/datajoint/diagram.py` — `Diagram.cascade`, `_propagate_restrictions`, `_apply_propagation_rule`, `_apply_propagation_rule_upward`, `_propagate_part_to_master`. -- Source: `src/datajoint/dependencies.py` — dependency-graph construction, `extract_master`. -- Issue: [datajoint-python #1429](https://github.com/datajoint/datajoint-python/issues/1429) — bug report and motivating examples for the upward propagation rules. -- [Master-Part Specification](master-part.md) — Part-Master contract. -- [Diagram Specification](diagram.md) — graph operations on the dependency graph. -- [Data Manipulation Specification](data-manipulation.md) — `delete()` and `delete_quick()` user-facing API. +- Source: `src/datajoint/diagram.py` (`Diagram.expand`, `Diagram.cascade`), `src/datajoint/dependencies.py` (dependency graph, `topo_sort`, `extract_master`). +- Issues: [#1429](https://github.com/datajoint/datajoint-python/issues/1429) (part-to-master lift with renamed foreign keys and part-of-part chains), [#1496](https://github.com/datajoint/datajoint-python/issues/1496) (delete-time materialization), [#1589](https://github.com/datajoint/datajoint-python/issues/1589), [#1590](https://github.com/datajoint/datajoint-python/issues/1590), [#1591](https://github.com/datajoint/datajoint-python/issues/1591) (rows deleted outside the cascade). +- [Diagram Specification](diagram.md#traversal-specification) — the definitions of `D`, `D1`, and the computation. +- [Traversal Shape Catalog](traversal-shapes.md) — worked examples. +- [Master-Part Specification](master-part.md) — the master-part contract. +- [Data Manipulation Specification](data-manipulation.md) — `delete()` and `delete_quick()`. diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 89c94729..4f5c0e7b 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -1,10 +1,10 @@ # Diagram Specification -Schema visualization as directed acyclic graphs. +Schema visualization as directed acyclic graphs, and traversal of the rows they connect. ## Overview -`dj.Diagram` visualizes DataJoint schemas as directed graphs showing tables and their foreign key relationships. It provides multiple output formats including SVG, PNG, and Mermaid syntax. +`dj.Diagram` visualizes DataJoint schemas as directed graphs showing tables and their foreign key relationships. It provides multiple output formats including SVG, PNG, and Mermaid syntax. It also computes, for a restricted table, the rows that depend on it or that it depends on: the rows a delete must remove, the inputs of a `make()`, or a self-contained export. ## Design Principles @@ -12,6 +12,7 @@ Schema visualization as directed acyclic graphs. 2. **Graph algebra**: Combine and filter diagrams with set operators 3. **Visual encoding**: Table tiers distinguished by shape and color 4. **Flexible layout**: Configurable direction and schema grouping +5. **One traversal**: `cascade` and `trace` are the two directions of `expand`, defined once in the [Traversal specification](#traversal-specification) --- @@ -92,6 +93,9 @@ dj.Diagram(ComputedTable) - 10 **Note:** Order matters. `diagram + 1 - 1` may differ from `diagram - 1 + 1`. +!!! version-added "New in 2.4" + On a diagram returned by `expand()`, `cascade()`, or `trace()`, only `d1 + d2` of two diagrams with the same direction is defined; the other operators raise. See [Expanded diagrams](#expanded-diagrams). + ### Collapsing Schemas !!! version-added "New in 2.1" @@ -122,106 +126,109 @@ dj.Diagram(Subject) + dj.Diagram(analysis).collapse() !!! version-added "New in 2.2" Operational methods (`Diagram.cascade()`, `restrict`, `counts`, `prune`) were added in DataJoint 2.2. -Diagrams can propagate restrictions through the dependency graph and inspect affected data using the graph structure. These methods turn Diagram from a visualization tool into a graph computation and inspection component. All mutation operations (delete, drop) are executed by `Table.delete()` and `Table.drop()`, which use Diagram internally. +!!! version-added "New in 2.4" + `Diagram.expand()` is the one traversal behind `cascade()` and `trace()`. Its rows are defined in the [Traversal specification](#traversal-specification) below. -### `Diagram.cascade()` (class method) +A diagram can compute, for a restricted table, the rows a delete must remove, the rows a `make()` may read, or the rows an export must carry. The methods in this section compute and inspect those rows without changing any data. `Table.delete()` and `Table.drop()` use them internally and carry out the mutation. + +### `Diagram.expand()` (class method) ```python -dj.Diagram.cascade(table_expr, part_integrity="enforce") +dj.Diagram.expand(seed, direction="down") ``` -Create a cascade diagram for delete. Builds a complete dependency graph from the table expression, includes all descendants across all loaded schemas, propagates the restriction downstream using **OR** semantics — a descendant row is marked for deletion if *any* ancestor path reaches it — and **trims** to the cascade subgraph. - | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `table_expr` | QueryExpression | — | A restricted table expression (e.g., `Session & 'subject_id=1'`) | -| `part_integrity` | str | `"enforce"` | Master-part integrity policy | - -**Returns:** New `Diagram` containing only the seed table and its descendants, with cascade restrictions applied. +| `seed` | QueryExpression | — | One table with any restriction (see [Seeds](#seeds)) | +| `direction` | str | `"down"` | `"down"`, `"up"`, or `"both"` | -**`part_integrity` values:** +**Returns:** a `Diagram` whose nodes are the tables the expansion reaches, each holding the rows of that table in the expansion. -| Value | Behavior | -|-------|----------| -| `"enforce"` | Default. The preview itself never errors; master-part integrity is enforced by `Table.delete()`'s post-check, which rolls back the delete if part rows would be removed without their masters (see [Cascade Spec](cascade.md#part_integrity-modes)). | -| `"ignore"` | Allow deleting parts without masters | -| `"cascade"` | Propagate restriction upward from part to master, then re-propagate downstream to all sibling parts | +| `direction` | Rows | Used for | +|---|---|---| +| `"down"` | `D(S)`: the seed rows and every row that depends on them | delete, impact analysis | +| `"up"` | `U(S)`: the seed rows and every row they depend on | the inputs of `make()`, lineage | +| `"both"` | `U(D(S))`: a region around the seed that is consistent on its own | export | -With `"cascade"`, the restriction flows **upward** from a part table to its master: the restricted part rows identify which master rows are affected, those masters receive a restriction, and that restriction propagates back downstream through the normal cascade — deleting the entire compositional unit (master + all parts), not just the originally matched part rows. +`S` is the set of rows the seed selects. `D`, `U`, and `U(D(S))` are defined under [Operations](#operations). ```python -# Preview cascade impact across all loaded schemas -dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts() -``` +# Every row that depends on subject 1, by table +dj.Diagram.expand(Subject & {"subject_id": 1}, "down").counts() -`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. +# Everything a selected session was derived from +dj.Diagram.expand(Session & {"subject_id": 1, "session_id": 2}, "up")[Subject] +``` -### `Diagram.trace()` (class method) +### `Diagram.cascade()` (class method) ```python -dj.Diagram.trace(table_expr) +dj.Diagram.cascade(table_expr, part_integrity="enforce") ``` -The **upstream mirror of `cascade()`**. Where `cascade()` walks downstream to every descendant, `trace()` walks **upstream** from a (possibly restricted) table expression to every ancestor across all loaded schemas, propagating the restriction along the way. Like `cascade()`, convergence is **OR** — an ancestor is included if reachable through *any* FK path from the seed. +The preview of `(table_expr).delete(part_integrity=...)`. It loads every schema downstream of the seed and holds the rows the delete would remove. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `table_expr` | QueryExpression | — | A restricted table expression whose ancestry is traced | +| `table_expr` | QueryExpression | — | The seed, e.g. `Session & {"subject_id": 1}` | +| `part_integrity` | str | `"enforce"` | Master-part integrity policy | -**Returns:** New `Diagram` containing the seed and its ancestors, each pre-restricted through the FK join path. +| `part_integrity` | Rows | +|---|---| +| `"cascade"` | `D(S)`, the same rows as `expand(table_expr, "down")` | +| `"ignore"` | `D1(S)`: the rows reached through foreign keys alone, with part tables treated as plain tables | +| `"enforce"` (default) | `D1(S)`. `Table.delete()` deletes these rows only if they include the master of every part row among them, and otherwise raises before deleting anything | -Access a pre-restricted ancestor by indexing the trace: +With `"enforce"`, the preview holds `D1(S)` and does not raise; the check runs in `Table.delete()`. `part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. The [Cascade Specification](cascade.md) covers the delete itself. + +!!! version-changed "Changed in 2.4" + `"enforce"` checks each part row's own master, before any row is deleted. In 2.3 it was a table-level check run after the delete. With `"cascade"`, a master joins the delete only if one of its part rows did ([#1591](https://github.com/datajoint/datajoint-python/issues/1591)). ```python -trace = dj.Diagram.trace(MyChild & key) -session = trace[Session] # Session restricted to ancestors of MyChild & key -session.fetch1('session_date') +# Preview the impact of a delete across all loaded schemas +dj.Diagram.cascade(Session & {"subject_id": 1}).counts() ``` -`trace()` reuses the same **upward propagation rules** (U1/U2/U3) documented in the [Cascade Spec](cascade.md#upward-propagation-child-parent). The [Upstream Trace Specification §1](trace.md#1-diagramtracetable_expr) is the normative spec for its API and semantics; inside `make()`, `self.upstream` is the per-`key` instance of a trace. - -### `restrict()` +### `Diagram.trace()` (class method) ```python -diag.restrict(table_expr) +dj.Diagram.trace(table_expr) ``` -Select a subset of data for export or inspection. Starting from a restricted table expression, propagate the restriction downstream through all descendants using **AND** semantics — a descendant row is included only if *all* restricted ancestors match. The full diagram is preserved (ancestors, unrelated tables) so that `restrict()` can be called again from a different seed table, building up a multi-condition subset incrementally. +The rows `table_expr` depends on: `U(S)`, the same rows as `expand(table_expr, "up")`. It loads every schema upstream of the seed. The [Upstream Trace Specification](trace.md) covers its use and `self.upstream`, the per-`make()` instance of a trace. -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `table_expr` | QueryExpression | — | A restricted table expression | - -**Returns:** New `Diagram` with restrict conditions applied. The graph is not trimmed. +```python +trace = dj.Diagram.trace(MyChild & key) +trace[Session].fetch1("session_date") +``` -**Constraints:** +!!! version-changed "Changed in 2.4" + A trace includes all parts of every master it reaches, and the ancestors of those parts ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). In 2.3 it included a part only if the part lay on a foreign-key path to the seed. -- **Chainable** — call multiple times to add conditions from different seed tables -- Cannot be called on a Diagram produced by `Diagram.cascade()` -- `table_expr.full_table_name` must be a node in the diagram +### Indexing: `d[T]` ```python -# Chain multiple restrictions (AND semantics) -diag = dj.Diagram(schema) -restricted = (diag - .restrict(Subject & {'species': 'mouse'}) - .restrict(Session & 'session_date > "2024-01-01"')) +d[Session] # a Table class or instance +d["Session"] # a class name, or a full table name ``` +Returns table `T` restricted to its rows in the expansion, as a query expression, in every direction. It raises `DataJointError` if `T` is not a node of the diagram. A node can hold zero rows; indexing it returns an empty expression. + +!!! version-changed "Changed in 2.4" + In 2.3, indexing worked only on diagrams returned by `trace()`. + ### `counts()` ```python diag.counts() ``` -Return affected row counts per table without modifying data. Works with both `cascade()` and `restrict()` restrictions. +Returns the number of rows each node holds, without modifying data. -**Returns:** `dict[str, int]` — mapping of full table names to affected row counts. - -**Requires:** `Diagram.cascade()` or `restrict()` must be called first. +**Returns:** `dict[str, int]` — full table name to row count. Tables in the diagram with no rows map to 0. ```python -counts = dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts() +counts = dj.Diagram.cascade(Session & {"subject_id": 1}).counts() # {'`lab`.`session`': 3, '`lab`.`trial`': 45, '`lab`.`processed_data`': 45} ``` @@ -231,23 +238,15 @@ counts = dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts() diag.prune() ``` -Remove tables with zero matching rows from the diagram view. This only affects the diagram object — no tables or data are modified in the database. Without prior restrictions, removes physically empty tables from the diagram. After `restrict()`, removes tables where the restricted query yields zero rows. - -**Returns:** New `Diagram` with empty tables removed. +Removes the nodes that hold zero rows from the diagram. No table or data is modified. On a diagram that was not expanded, it removes physically empty tables. -**Constraints:** Cannot be used on a Diagram produced by `Diagram.cascade()`. Cascade diagrams must retain all descendant tables because a table empty at cascade time could have rows by the time `delete()` executes. - -**Note:** Queries the database to determine row counts. The underlying graph structure is preserved — subsequent `restrict()` calls can still seed at any table in the schema. +**Constraints:** `prune()` raises on a `down` diagram, including any diagram returned by `cascade()`. A delete must keep every table it reaches, because a table that is empty when the diagram is built can gain rows before the delete runs. `prune()` is allowed on `up` and `both` diagrams. ```python -# Export workflow: restrict, prune, visualize -export = (dj.Diagram(schema) - .restrict(Subject & {'species': 'mouse'}) - .restrict(Session & 'session_date > "2024-01-01"') - .prune()) - -export.counts() # only tables with matching rows -export # visualize the export subgraph +# Export region around one subject, without empty tables +region = dj.Diagram.expand(Subject & {"subject_id": 1}, "both").prune() +region.counts() +region # display the region ``` ### Iteration @@ -259,34 +258,291 @@ Diagrams support iteration in topological order: | `for ft in diagram` | Parents first | Data export, inspection | | `for ft in reversed(diagram)` | Leaves first | Cascade delete, drop | -Each iteration yields a `FreeTable` with any cascade or restrict conditions applied. Only nodes in the diagram's visible set (`nodes_to_show`) are yielded. +Each iteration yields a `FreeTable` restricted to the node's rows. Only nodes in the diagram's visible set (`nodes_to_show`) are yielded. `Table.delete()` and `Table.drop()` use `reversed(diagram)` internally to execute mutations in safe dependency order. -### Restriction Propagation +### `restrict()` + +!!! version-deprecated "Deprecated in 2.4, removed in 3.0" + `Diagram.restrict()` emits a `DeprecationWarning`. Its behavior is frozen. Use `Diagram.expand()`; see [Retiring `restrict`](#retiring-restrict) for the migration. + +```python +diag.restrict(table_expr) +``` + +Restricts a full diagram for export: the restriction propagates downstream, and where several restricted ancestors meet, a row is kept only if it matches all of them (AND). The full diagram is preserved, including ancestors and unrelated tables, so `restrict()` can be chained from other seeds. It cannot be called on a diagram produced by `cascade()`. + +### Unloaded schemas + +`expand()`, `cascade()`, and `trace()` first load every schema reachable through foreign keys in the direction of travel, including schemas the current process has not activated ([X1](traversal-shapes.md#x1-unloaded-downstream-schema)). If a delete still fails with a foreign-key error, DataJoint names the two likely causes: a downstream schema the connection cannot see, or a row inserted concurrently. + +--- + +## Traversal specification + +!!! version-added "New in 2.4" + This section is the normative definition of `Diagram.expand()`, `cascade()`, and `trace()`. The test suite checks the engine against an executable copy of these definitions, and the [Traversal Shape Catalog](traversal-shapes.md) works through one example per schema shape. + +The definitions are about rows; the computation is about tables and restrictions. + +### Base principles + +- **P1, referential integrity.** Every foreign-key value names an existing parent row. +- **P2, compositional integrity.** A master row and its part rows are one entity. They are inserted together and deleted together. +- **P3, reproducibility.** `make(key)` reads only rows that `key` depends on. +- **P4, minimality.** An operation touches no row it does not have to. + +### Model and assumptions + +**Foreign-key references.** A foreign key `e` of a child table `C` references a parent table `P`. Its attribute map `attr_map_e` pairs attributes of `C` with the primary-key attributes of `P`; the names differ when the declaration renames them with `.proj()`. For a row `c` of `C`, `ref_e(c)` is the row of `P` whose primary key equals `c`'s values on those attributes. By P1, that row exists. + +**A2, NULLs.** If any of `c`'s attributes in `e` is NULL, `ref_e(c)` is undefined: `c` references nothing through `e`. This is SQL's MATCH SIMPLE rule, which both supported backends apply: a foreign key with a NULL column is not checked. Foreign keys can share columns ([S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column)), so a row can have some columns of a foreign key set and others NULL. Such a row still references nothing through that foreign key. + +**Parallel foreign keys.** Two foreign keys between the same pair of tables are separate references, each with its own `attr_map` ([S5](traversal-shapes.md#s5-parallel-foreign-keys)). + +**A1, owned parts.** A part is **owned** by its master if a foreign key in the part's primary key references the master, or references an owned part of the same master. A foreign key is in the primary key when all its attributes are primary-key attributes of the part, wherever the reference is declared. The references along this chain may rename attributes. A foreign key in the primary key cannot be NULL, so every row of an owned part references a row at each step of the chain. A master and its owned parts form the master's **group**. + +The **owning attributes** of an owned part are the part's attributes that the chain maps onto the master's primary key: + +- for `-> master` in the primary key, the master's own key attributes; +- for `-> master.proj(m_id="master_id")`, the attribute `m_id`; +- for a reference to an owned part, the attributes that the reference maps onto that part's owning attributes ([G7](traversal-shapes.md#g7-owned-part-of-part-with-renamed-attributes)). + +If a part has several owning chains, its owning attributes come from an unrenamed `-> master` if it has one, and otherwise from the owning reference whose attributes come first in the part's heading, which follows the order of the definition. + +For a row `r` of an owned part: + +- `master(r)` is the master row whose primary key equals `r`'s owning attributes; +- `inst(r)`, the **instance** of `r`, is `master(r)` together with every row of every owned part of the same master whose owning attributes have the same values. + +For a master row `m`, `inst(m)` is `m` together with the rows of its owned parts whose owning attributes equal `m`'s key. For any other row `r`, `inst(r) = {r}`. + +A part that is not owned gets a warning at declaration, and it is a plain table in traversal, in `topo_sort`, and in the declaration check: it has no master and no instance. This covers a part that does not reference its master, one that references it only through a foreign key with attributes outside the primary key (a secondary or nullable reference), and one that references it only through a part that is not owned ([G6](traversal-shapes.md#g6-a-part-its-master-does-not-own)). + +!!! version-added "New in 2.4" + Ownership decides which parts belong to a master's group ([#1619](https://github.com/datajoint/datajoint-python/issues/1619)). In 2.3, every part was grouped with its master by its table name. + +**Row relations.** For a row `r`: + +- `child(r) = {c : ref_e(c) = r for some foreign key e}` — the rows that reference `r`; +- `parent(r) = {ref_e(r) : e a foreign key of r's table, ref_e(r) defined}` — the rows `r` references; +- `down(r) = child(r) ∪ inst(r)`; +- `up(r) = parent(r) ∪ inst(r)`. + +`up` is exactly the converse of `down`: `y ∈ down(x)` if and only if `x ∈ up(y)`. `parent` is the converse of `child`, and sharing an instance is symmetric. + +### Operations + +A set of rows `X` is **closed under** a relation `R` when, for every row `x` in `X`, all of `R(x)` is in `X` too. The **closure** of a set `S` under `R` is the smallest set that contains `S` and is closed under `R`. It is computed by starting from `S` and adding `R`-neighbors until nothing new appears; in mathematical terms it is the *least fixed point* of that step. Every closure here is finite because the database is. + +Let `S` be the set of rows the seed selects. Then: + +- `D(S)` is the closure of `S` under `down`; +- `U(S)` is the closure of `S` under `up`; +- `D1(S)` is the closure of `S` under `child` alone, with no instance step. + +| Call | Rows | +|---|---| +| `expand(seed, "down")` | `D(S)` | +| `expand(seed, "up")` | `U(S)` | +| `expand(seed, "both")` | `U(D(S))` | + +**Why `both` is `U(D(S))`.** An export region must be importable on its own: every foreign key in it must resolve inside it, and every instance in it must be whole. `U(D(S))` is the smallest such region that contains everything derived from `S` (T6). The other two combinations fail: -When `cascade()` or `restrict()` propagates a restriction from a parent table to a child table, one of three rules applies depending on the foreign key relationship: +- `D(U(S))` adds every descendant of every ancestor. For a seed under a shared lookup table, that is most of the database. +- `D(S) ∪ U(S)` misses the other parents of the descendants. In [S5](traversal-shapes.md#s5-parallel-foreign-keys), exporting neuron 1 this way would carry synapse (3, 1) without neuron 3. -**Rule 1 — Direct copy:** When the foreign key is non-renamed and the restriction attributes are a subset of the child's primary key, the restriction is copied directly to the child. +**Delete.** `Table.delete(part_integrity=...)` and its preview `Diagram.cascade()`: -**Rule 2 — Renamed projection:** When the foreign key uses attribute renaming (e.g., `subject_id` → `animal_id`), the parent is projected with the attribute mapping to match the child's column names. +| `part_integrity` | Deletes | +|---|---| +| `"cascade"` | `D(S)` | +| `"ignore"` | `D1(S)`. A part row can be deleted while its master and sibling part rows stay. | +| `"enforce"` (default) | `D1(S)`, if it is **lift-closed**: it contains `master(r)` for every part row `r` in it. Otherwise it raises **before deleting anything**, naming a part row and its master. | -**Rule 3 — Full projection:** When the foreign key is non-renamed but the restriction uses attributes not in the child's primary key, the parent is projected (all attributes) and used as a restriction on the child. +By T7, `enforce` deletes `D(S)` when the group rule would add nothing to `D1(S)`, and refuses otherwise. -**Convergence behavior:** +**Trace.** `Diagram.trace(seed)` selects `U(S)`. -When a child table has multiple restricted ancestors, the convergence rule depends on the mode: +**`self.upstream`.** `make(key)` runs before the row for `key` is inserted, so `self & key` selects nothing, and `U(∅) = ∅` would leave `self.upstream` empty. `self.upstream` therefore starts from the parents that row will have. For each foreign key `e` of the table that lies in its primary key, `relabel_e(key)` renames `key`'s values onto the primary key of the parent `P_e` through `attr_map_e`, and -- **`cascade()` (OR):** A child row is affected if *any* path from a restricted ancestor reaches it. This is appropriate for delete — if any reason exists to delete a row, it should be deleted. -- **`restrict()` (AND):** A child row is included only if *all* restricted ancestors match. This is appropriate for export — only rows satisfying every condition are selected. +``` +self.upstream = ⋃ over primary foreign keys e of expand(P_e & relabel_e(key), "up") +``` + +Foreign keys outside the primary key are left out: their values are not known before the insert. By T2, `self.upstream` is `U` of the parents of the row being computed. + +!!! version-changed "Changed in 2.4" + In 2.3, `self.upstream` was `Diagram.trace(self & key)`. It returned the parents only because the engine copied the key's condition upward, which is not sound for other seeds (see [R1](#r1-the-edge-rule)). + +### Derived results + +Each result below follows from the definitions and is checked by the test suite. + +**T1, closure.** `D`, `U`, and `D1` are *extensive* (`X(S) ⊇ S`), *monotone* (`A ⊆ B` implies `X(A) ⊆ X(B)`), and *idempotent* (`X(X(S)) = X(S)`). + +Every closure under a relation has these properties: the result contains `S` by construction, a larger starting set reaches at least the same rows, and a set that is already closed gains nothing from closing it again. + +**T2, additivity.** `X(A ∪ B) = X(A) ∪ X(B)` for `X` = `D`, `U`, `D1`, and for `both`. + +Each step adds the neighbors of a single row, so a row is in `X(S)` exactly when a chain of steps leads to it from one row of `S`, and that is the case for `A ∪ B` exactly when it is the case for `A` or for `B`. For `both`, `U(D(A ∪ B)) = U(D(A) ∪ D(B)) = U(D(A)) ∪ U(D(B))`. So several seeds give the union of their expansions: `expand` never needs an intersection, `d1 + d2` of the same direction is exact, and `self.upstream` is built as a union. + +**T3, duality.** `y ∈ D({x})` if and only if `x ∈ U({y})`. + +A chain of `down` steps from `x` to `y`, read backwards, is a chain of `up` steps from `y` to `x`, because `up` is the converse of `down`. So deleting `x` invalidates exactly the rows whose upstream contains `x`. Because a delete lifts part rows to their master, a trace must include the parts of an ancestor master, and their ancestors ([#1481](https://github.com/datajoint/datajoint-python/issues/1481); [G5](traversal-shapes.md#g5-merge-table)). `ignore`'s `D1` is likewise dual to a trace without the instance step. + +**T4, delete safety.** If the database satisfies P1, removing `D(S)` keeps P1 and removes each instance either whole or not at all (P2). `D(S)` is the smallest superset of `S` with both properties (P4). + +`D(S)` contains every child of every row in it, so no remaining row references a removed one; and it contains the instance of every row in it, so no instance is split. Conversely, a removed set `X ⊇ S` that keeps P1 must contain the children of its rows, and one that keeps instances whole must contain their instances. Such an `X` is closed under `down` and so contains `D(S)`. + +**T5, read boundary.** `U(S)` is consistent on its own: every foreign key of its rows resolves inside it, and every instance in it is whole. It is the smallest superset of `S` with that property, and so the set of rows `S` depends on (P3). + +The argument is T4's with `parent` in place of `child`. + +**T6, `both`.** `U(D(S))` contains `D(S)`, is closed under `up`, and so is consistent on its own and contains every instance it touches. It is the smallest set with these properties that contains `D(S)`. `both` is extensive and monotone but not idempotent. + +`U(D(S))` is `U` of `D(S)`, so T5 applies to it with `D(S)` as the seed. For non-idempotence, take a table `A` with two child tables `B` and `C`, and a row `a` with one child `b` in `B` and one child `c` in `C`. With `S = {b}`, `both(S) = U({b}) = {a, b}`, and `both({a, b}) = U({a, b, c}) = {a, b, c}`: the second application picks up `a`'s other descendant. + +**T7, enforce.** `D1(S)` is lift-closed if and only if `D1(S) = D(S)`. + +Every row of an owned part can be reached from its master row by `child` steps along its owning chain: each reference in the chain lies in a primary key, so it is never NULL, and it maps owning attributes onto owning attributes, so it stays in the same instance. So if `D1(S)` contains the master of each of its part rows, it contains their whole instances, is closed under `down`, and equals `D(S)`. Conversely, `D(S)` is closed under `inst`, so if `D1(S) = D(S)` it is lift-closed. `enforce` therefore deletes only when the group rule would add nothing. + +**T8, ordering.** `topo_sort` succeeds if and only if the group quotient `Q` is acyclic. The **group quotient** `Q` is the table-level foreign-key graph with each master and its owned parts merged into a single node; a part that is not owned stays a node of its own. + +`topo_sort` keeps each owned part right after its master, so it sorts the table graph after adding, for each owned part, an edge from each of the part's parents outside its group to its master. The master reaches each owned part along its owning chain, so a cycle in `Q` becomes a cycle in that graph. Conversely, a cycle in that graph cannot stay inside one group, because foreign keys between tables never form a cycle, so it maps to a cycle in `Q`. + +The declaration check keeps `Q` acyclic ([#1619](https://github.com/datajoint/datajoint-python/issues/1619); [X2](traversal-shapes.md#x2-group-cycle-refused-at-declaration)). An owned part may reference its master, an owned part of the same master, or a table that does not depend on its master, with groups taken whole; parts the master does not own count as outside the group. A reference to any other table would close a cycle in `Q`, and the declaration is refused before any table is created. + +### Computation + +The engine computes the rows above as restrictions on tables, without reading the rows, except where noted. In the formulas below, a projection `.proj(...)` keeps only the attributes it lists, renamed as shown, so no other attribute of the source takes part in the match. + +#### R1, the edge rule + +**Down.** Crossing a foreign key `e` from a restricted parent `P` to the child `C`, the child is restricted by the restricted parent, projected onto `P`'s primary key, with each key attribute renamed to its name in `C` through `attr_map_e`: + +``` +C & P_restricted.proj() +``` + +The parent's condition may instead be **copied** to the child unchanged, but only when the attributes it names are among `e`'s attributes, `e` is not renamed, and none of `e`'s columns is nullable. Here, the attributes a condition names are every key of a dict, including keys that are not attributes of the parent, and the whole heading of a query expression. Under these conditions each child row holds the same values under the same names as the parent row it references, and it references one, so the copied condition selects exactly the children. [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column) shows the nullable case, [S4](traversal-shapes.md#s4-renamed-foreign-key) the renamed case, and [S7](traversal-shapes.md#s7-diamond-reached-through-a-secondary-foreign-key) a condition on an attribute the child holds through a different parent. + +A copy is a shortcut, and it must select the same rows as the restriction. Two kinds of condition would change meaning if moved, and the rule above already keeps them from being copied: + +- a dict with keys that are not attributes of the parent: they are ignored on the parent but can match attributes of the child; +- a query expression that shares more attributes with the child than with the parent: a restriction by a query expression matches on all shared attributes. Copied to `Session`, the seed condition of [K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant), `Session & "quality = 0"`, would match on every attribute of `Session` and select only the sessions of quality 0. Copied the same way, the seed `Subject - (Session & "quality = 0")` would select every session of another quality, including sessions of subjects that are not deleted. + +!!! note + The 2.3 engine copies the compiled SQL predicate, which reads only the parent's columns. That is why 2.3 copies dict conditions, and conditions given as query expressions, correctly. (A seed restricted by a query expression failed in 2.3 for a different reason, before any copy; 2.3.5 replaces such a seed by its keys, [#1610](https://github.com/datajoint/datajoint-python/issues/1610).) The 2.4 plan refers to conditions rather than to compiled SQL, so the rule above applies. + +**Parallel foreign keys** combine with OR: a child row is reached if any of its foreign keys references a restricted row. + +**Up.** Crossing `e` from a restricted child `C` to the parent `P` is always the restriction -**Multiple foreign keys to the same parent:** +``` +P & C_restricted.proj() +``` + +This selects the parent rows referenced by at least one restricted child row. It is never a copy: a condition on the child describes child rows, and copied to the parent it would select parents that no restricted child references ([S2](traversal-shapes.md#s2-secondary-foreign-key), [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column)). + +The one shortcut upward applies to keys materialized from the child table itself (for example `(C & cond).keys()`) that include every column of `e`: these may be relabeled through `attr_map_e`. They name existing child rows, so the relabeled keys name exactly the referenced parents. A user-supplied key dict may name rows that do not exist, so it crosses upward by restriction too ([K5](traversal-shapes.md#k5-empty-seeds)). + +#### R2, the group rule + +For a master `M` and its part `Pi`, with `R(T)` the restriction on table `T`: + +- **lift:** `M & R(Pi).proj(m1="o1", m2="o2", …)` — the masters of the restricted part rows. Each owning attribute `o` of `Pi` is renamed to the attribute `m` of the master's key that it maps onto; for an unrenamed `-> master` this is `M & R(Pi).proj()`. Every owned part row holds its owning attributes, so the lift is one step, with no walk along the foreign keys between parts ([G2](traversal-shapes.md#g2-part-of-part-with-two-chains), [G7](traversal-shapes.md#g7-owned-part-of-part-with-renamed-attributes)). The part's other references to its master are not used for the lift ([G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master)). +- **re-expand:** `Pi & R(M)` — every part row of the restricted masters. + +The lift is existential: a master joins only if at least one of its part rows did ([G1](traversal-shapes.md#g1-part-to-master-lift)). + +#### One pass over the group quotient + +Because `Q` is acyclic (T8), the engine visits each node of `Q` once: in `topo_sort` order for `down`, and in reverse order for `up`. + +- A **plain table** `T` gets `R(T)` = its seed ∪ R1 from each restricted neighbor on the incoming side (its parents for `down`, its children for `up`). +- A **group** with master `M` gets `R(M)` = the seed on `M` ∪ R1 into `M` ∪ lift(R1 into each part ∪ the seed on each part). Then each part `Pi` gets `Pi & R(M)`. + +`up` is the same pass with every edge reversed. The pass is exact because every neighbor on the incoming side is final before it is used (`Q` is acyclic), and because every foreign key inside a group maps owning attributes onto owning attributes, so steps inside a group never leave the instances already in `R(M)`. The next subsection covers the groups where that fails. + +#### Groups that iterate + +A group iterates if and only if some foreign key inside it does not map the referencing part's owning attributes onto the referenced table's owning attributes (for the master, its primary key). Such a foreign key links different instances of the group. In [G4](traversal-shapes.md#g4-renamed-sibling-foreign-key-chained-across-three-instances), a `Carry` row of run 2 references a `Step` row of run 1 through `prev_run`, which is not an owning attribute of `Carry`; in [G3](traversal-shapes.md#g3-a-second-renamed-foreign-key-to-its-own-master), a part references another row of its own master through `other_session`. Renaming the master's key is one way this happens. A foreign key references the whole primary key, which contains the referenced table's owning attributes, so every reference inside a group carries a master key; the question is only whether it is the referencing part's own. + +The engine detects such a foreign key from its `attr_map`. For that group only, it evaluates the lifted master keys and repeats the group step until no new key appears. This is the only place where a traversal reads rows before a delete. + +#### Seeds + +- A seed is one table with any restriction. Projections, joins, aggregations, and unions are refused as seeds. The restriction itself may use them, for example a subquery on a descendant ([K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant)) or on the seed's own table ([K2](traversal-shapes.md#k2-self-referencing-seed)). +- A `dj.Top` seed is first replaced by the primary keys it selects ([K4](traversal-shapes.md#k4-djtop-seed)). +- A dict that names none of the table's attributes means all rows, as in any restriction ([Query Algebra §2.4](query-algebra.md#24-dictionary-conditions)). As a seed, it raises a warning ([K6](traversal-shapes.md#k6-unrestricted-seed)). +- A materialized seed (a dict, a list of dicts, or `.keys()`) may cross an edge by **relabeling**: keeping only the keys the foreign key carries, renamed through `attr_map`. Relabeling is used only where it selects the same rows as R1. Downward, every key must name, among the parent's attributes, only attributes the foreign key carries; and if the foreign key has a nullable column, every key must name all of the foreign key's attributes, none of them NULL. Otherwise relabeling changes the rows: `Session & {"quality": 0}` relabels to `{}` across `Trial`'s foreign key and would select every trial, and on the schema of [S3](traversal-shapes.md#s3-nullable-foreign-key-sharing-a-primary-key-column), `Surgery & {"subject_id": 1, "surgery_id": None}` selects no surgery but, relabeled, would select session (1, 2; NULL). Upward, relabeling applies only to keys read from the child table that contain all of the foreign key's columns. A live seed — a string or a query expression — crosses by restriction. Both give the same rows ([K3](traversal-shapes.md#k3-seed-kinds-give-the-same-rows)). +- The attributes a string condition names are read from its text outside any subquery. + +#### The plan -When a child table references the same parent through multiple foreign keys (e.g., `source_mouse` and `target_mouse` both referencing `Mouse`), these paths always combine with **OR** regardless of the propagation mode. Each foreign key path is an independent reason for the child row to be affected — this is structural, not operation-dependent. +The engine first builds a **plan**: an intermediate form, not SQL. Its nodes are `Seed`, `Restrict(src, attr_map)`, `Lift(part)`, `Reexpand(master)`, `Or`, and `Literal`. A node refers to the restrictions it uses by a handle to another node; compiled SQL is not copied from one restriction into another, except that a copy under R1 applies the seed's own compiled condition to the child. The plan compiles to query expressions for the database. It can also be evaluated against rows held in memory, which is how the test suite compares it with the definitions above. + +#### Materialization happens only at delete time + +Traversal and preview (`counts()`, iteration, `d[T]`) stay symbolic, apart from [groups that iterate](#groups-that-iterate) and `dj.Top` seeds, which are replaced by their keys when the traversal starts. Before the first DELETE, the engine **materializes** a node, replacing its restriction by the literal keys it selects, when one of the node's local terms reads a table that is deleted no later than the node itself. A *local term* is the node's seed or a lift into it. The terms a node inherits through R1 from a parent are safe: `Table.delete()` deletes in reverse topological order, so a parent still holds its rows when its children are deleted. + +In practice: + +- every lift is materialized, because a part is deleted before its master; +- a seed is materialized unless its restriction is literal keys, because a string or query-expression restriction is SQL text whose reads cannot be inspected ([K1](traversal-shapes.md#k1-seed-restricted-by-a-descendant), [K2](traversal-shapes.md#k2-self-referencing-seed)); +- large key sets are applied in chunks. + +Without this, a restriction that reads a table emptied earlier in the same delete would match nothing, and its rows would stay behind ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)). + +#### Concurrency + +- Part rows are deleted as `Pi & R(M)`, with `M` materialized. A part row inserted by another session after that DELETE makes the master's DELETE fail on its foreign key (`ON DELETE RESTRICT`), so the transaction rolls back and the part row is never left without its master. +- The foreign-key error message mentions concurrent inserts. +- `enforce` runs the `down` plan once its check passes. By T7, its rows are `D1(S)`. +- The table-level check after the delete stays as a cheap backstop. + +### Expanded diagrams + +A diagram returned by `expand()`, `cascade()`, or `trace()` is an **expanded diagram**. It has a direction, and it obeys the following contract. + +- **Nodes.** The node set is the *table-level reach* of the seed's table: the tables reached by following foreign keys in the direction of travel, together with the master and all owned parts of every group that has a reached member. (For `D1`, the group step is left out.) For `both`, it is the upward reach of the downward reach. A node may hold zero rows. +- **`d[T]`** returns `T` restricted to its rows, in every direction, and raises `DataJointError` if `T` is not a node. +- **`counts()`** works, and **iteration** runs in topological order. +- **Copying** a diagram, `dj.Diagram(d)`, keeps its direction. +- **`d1 + d2`** of two diagrams with the same direction holds, for each table, the union of their rows (T2). It raises for diagrams of different directions, for an expanded diagram and one that is not expanded, and for two `cascade()` diagrams with different `part_integrity` modes. +- **`-`, `*`, `+ n`, and `- n`** raise. +- **`prune()`** raises for `down` and is allowed for `up` and `both`. + +!!! version-changed "Changed in 2.4" + In 2.3, copying a diagram dropped its direction, and `d1 + d2` silently dropped the right operand's restrictions. + +### Retiring `restrict` + +`Diagram.restrict()` is deprecated in 2.4: it emits a `DeprecationWarning`, its behavior is frozen, and it is removed in 3.0. It intersected the conditions that reached a table (AND), which can split an instance, keeping a master with only some of its parts ([#1501](https://github.com/datajoint/datajoint-python/issues/1501)). `expand` takes unions instead, so its results are larger: they hold everything the seed reaches, and never part of an instance. + +| Use of `restrict` | Replacement | +|---|---| +| One condition, for export | `dj.Diagram.expand(Session & cond, "both")` | +| One condition, for impact | `dj.Diagram.expand(Session & cond, "down")` | +| Several conditions that share a descendant | Seed the descendant with all of them, e.g. `dj.Diagram.expand(Session & (Subject & cond_a) & cond_b, "both")` | +| Per-table AND across several seeds | The recipe below | + +The recipe intersects, table by table, the rows that separate `down` expansions reach: + +```python +seeds = [Subject & {"species": "mouse"}, Session & "session_date > '2024-01-01'"] +carved = {} +for diagram in (dj.Diagram.expand(seed, "down") for seed in seeds): + for table in diagram: # restricted FreeTable, parents first + name = table.full_table_name + carved[name] = carved[name] & table.proj() if name in carved else table +``` -**Unloaded schemas:** +Each `carved[name]` holds the rows that every seed reaching that table reaches. Unlike `restrict`, a single seed's paths combine by union. The result is not closed: it can split an instance and need not be importable on its own, so use it for inspection, and seed a common descendant for export. -If a descendant table lives in a schema that hasn't been activated (loaded into the dependency graph), the graph-driven delete won't know about it. The final `DELETE` on the parent will fail with a foreign key error. DataJoint catches this and produces an actionable error message identifying which schema needs to be activated. +The [Traversal Shape Catalog](traversal-shapes.md) gives the exact rows for each schema shape the specification distinguishes. --- @@ -473,7 +729,7 @@ combined = dj.Diagram.from_sequence([schema1, schema2, schema3]) ## Dependencies -Operational methods (`cascade`, `restrict`, `counts`, `prune`) use `networkx`, which is always installed as a core dependency. +Operational methods (`expand`, `cascade`, `trace`, `counts`, `prune`) use `networkx`, which is always installed as a core dependency. Diagram **visualization** requires optional dependencies: @@ -488,6 +744,10 @@ If visualization dependencies are missing, `dj.Diagram` displays a warning and p ## See Also - [How to Read Diagrams](../../how-to/read-diagrams.ipynb) +- [Traversal Shape Catalog](traversal-shapes.md) — Worked examples of `expand`, one per schema shape +- [Cascade Specification](cascade.md) — The delete case of `expand` +- [Upstream Trace Specification](trace.md) — The upstream case of `expand` and `self.upstream` +- [Master-Part](master-part.md) — Master-part groups - [Delete Data](../../how-to/delete-data.md) — Cascade inspection and delete workflow - [What's New in 2.2](../../about/whats-new-22.md) — Motivation and design - [Data Manipulation](data-manipulation.md) — Insert, update, delete specification diff --git a/src/reference/specs/index.md b/src/reference/specs/index.md index ac7d6bb9..baba6662 100644 --- a/src/reference/specs/index.md +++ b/src/reference/specs/index.md @@ -98,6 +98,7 @@ Each specification follows a consistent structure: | [Primary Keys](primary-keys.md) | Table Declaration | [Design Primary Keys](../../how-to/design-primary-keys.md/) | [Entity Integrity](../../explanation/entity-integrity.md/) | | [Fetch API](fetch-api.md) | Query Operators | [Fetch Results](../../how-to/fetch-results.md/) | — | | [Diagram](diagram.md) | Table Declaration | [Read Diagrams](../../how-to/read-diagrams.ipynb/) | — | +| [Traversal Shape Catalog](traversal-shapes.md) | Diagram, Master-Part | [Delete Data](../../how-to/delete-data.md/) | — | **Key concepts:** Restriction (`&`, `-`), projection (`.proj()`), join (`*`), aggregation (`.aggr()`), union, universal set (`U()`), attribute lineage, schema visualization diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 8a9b78a5..3bb6b939 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -2,15 +2,15 @@ This document specifies two interlocking features that give DataJoint a first-class **upstream read surface** — a way to query, from any set of rows, the exact ancestor rows they were derived from: -1. **`Diagram.trace(table_expr)`** — the upstream mirror of `Diagram.cascade()`; walks the FK graph from a restricted table expression to every ancestor. +1. **`Diagram.trace(table_expr)`** — the upstream mirror of `Diagram.cascade()`; selects every row the seed's rows depend on. 2. **`self.upstream`** — a property on `AutoPopulate` tables that exposes the trace for the current `key` during `make()` execution. -The two are designed as a unit. `trace` is the underlying graph operation; `self.upstream` is the ergonomic surface for `make()`. Together they make the "read from declared upstream" convention easy to follow and to inspect — which is exactly what makes a `make()` reproducible and statically verifiable (see the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract)). +The two are designed as a unit. `trace` is the underlying graph operation; `self.upstream` is the ergonomic surface for `make()`. Together they make the "read from declared upstream" convention easy to follow and to inspect — which is what makes a `make()` reproducible and statically verifiable (see the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract)). !!! version-added "New in DataJoint 2.3" Introduced together. Existing code that does not use these features is unaffected. -For the related downstream operation, see [Cascade Specification](cascade.md). For `make()` itself, see [AutoPopulate Specification](autopopulate.md). +A trace is the `up` case of `Diagram.expand`: `Diagram.trace(seed)` equals `Diagram.expand(seed, "up")`. The rows it selects are defined once, in the [Diagram specification's Operations](diagram.md#operations); this page specifies the upstream-specific behavior and the `self.upstream` surface. For the downstream case, see the [Cascade Specification](cascade.md). For `make()` itself, see the [AutoPopulate Specification](autopopulate.md). ## Why this exists @@ -19,27 +19,28 @@ A computed row's reproducibility rests on the convention that `make(self, key)` 1. A way to **construct** the upstream view as a query object: `Diagram.trace()`. 2. A way to **expose** that view ergonomically inside `make()` so users naturally do the right thing: `self.upstream`. -Without (1), downstream tools that need row-level lineage (data-lineage viewers, CDC publishers, audit logs) have to reimplement the FK walk themselves — repeatedly, with subtly different bugs. Without (2), even users who follow the convention write awkward boilerplate (`(Upstream & key).fetch1(...)` everywhere). +Without (1), downstream tools that need row-level lineage (data-lineage viewers, CDC publishers, audit logs) have to reimplement the foreign-key walk themselves — repeatedly, with subtly different bugs. Without (2), even users who follow the convention write awkward boilerplate (`(Upstream & key).fetch1(...)` everywhere). ## Concepts ### Trace as the upstream mirror of cascade -`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"*. The two share the same dependency graph, the same edge model, and most of the same propagation machinery — only the direction differs. +`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"* They are the two directions of one traversal, `expand`, over the same dependency graph and the same rules: -| Method | Direction | Convergence | Question answered | +| Method | `expand` direction | Rows | Question answered | |---|---|---|---| -| `Diagram.cascade(expr)` | downstream | OR — any FK path taints | What's affected if these rows are deleted? | -| `Diagram.restrict(expr)` | downstream | AND — must satisfy all FK paths | What satisfies all of these conditions? | -| `Diagram.trace(expr)` | **upstream** | **OR** — any FK path contributes | What contributed to these rows? | +| `Diagram.cascade(expr, part_integrity="cascade")` | `down` | `D(S)` | What's affected if these rows are deleted? | +| `Diagram.trace(expr)` | `up` | `U(S)` | What contributed to these rows? | -`trace` uses OR convergence because an ancestor entity contributes to a child row if it appears via *any* FK path. (An AND-flavored upstream analog — "ancestors that contributed via *every* path" — is not provided in 2.3.) +`U(S)` is the smallest set of rows that contains the seed rows `S`, the rows each of its rows references through a foreign key, and the whole master-part instance of each of its rows. It is consistent on its own: every foreign key of its rows resolves inside it, and every instance in it is whole ([T5](diagram.md#derived-results)). -### Reusing the propagation primitives +The two directions are exact mirrors ([T3](diagram.md#derived-results)): a row `y` depends on a row `x` exactly when deleting `x` would delete `y`. Results combine by union: an ancestor contributes if it is reachable through *any* foreign-key path, and there is no intersection. -`trace` applies the **upward propagation rules** (`U1`, `U2`, `U3`) defined in the [Cascade Specification](cascade.md#upward-propagation-child-parent), which are the symmetric inverses of `cascade`'s forward rules. Renamed FKs (`.proj()`) are reversed via U2; Part-of-Part chains are walked through naturally; multiple foreign keys between the same pair of tables are each reversed independently. +### The rules upstream -This is why `trace` cannot ship before the upward primitives exist in the codebase. As of DataJoint 2.3, the primitives are in place (added with [#1429's cascade fix](https://github.com/datajoint/datajoint-python/pull/1468)), and `trace` is a direct consumer. +`trace` crosses each foreign key upward by **restriction**: the parent is restricted to the rows referenced by at least one restricted child row, `parent & child_restricted.proj(...)`, with renamed attributes mapped back through the foreign key's `attr_map`. A condition on the child is never copied to the parent, because it describes child rows ([R1](diagram.md#r1-the-edge-rule)). A child row whose foreign key has a NULL column references nothing through it. + +The group rule ([R2](diagram.md#r2-the-group-rule)) applies upward as it does downward: a row of an owned part brings its master and the master's owned part rows, and a master row brings its owned part rows. A part that its master does not own is a plain table ([A1](diagram.md#model-and-assumptions)). Parallel foreign keys are each crossed on their own. ### The `make()` read/write boundary @@ -49,11 +50,11 @@ DataJoint's `make(self, key)` convention has always been: Read from self's declared upstream dependencies; write to self (and its Parts). ``` -This is the **read/write boundary**: data flows in from declared ancestors and out to the target table and its Parts. Anything that crosses this boundary in another direction undermines reproducibility — a `fetch` from an undeclared table makes that dependency invisible to the FK graph; an `insert` into another table sneaks rows past it. (The full rule set is the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract).) +This is the **read/write boundary**: data flows in from declared ancestors and out to the target table and its Parts. Anything that crosses this boundary in another direction undermines reproducibility — a `fetch` from an undeclared table makes that dependency invisible to the foreign-key graph; an `insert` into another table sneaks rows past it. (The full rule set is the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract).) `self.upstream` makes the read side of this boundary the path of least resistance: -- **`self.upstream`** is the recommended read channel. Reading through it keeps every dependency visible to the FK graph and pre-restricted to the current `key`. +- **`self.upstream`** is the recommended read channel. Reading through it keeps every dependency visible to the foreign-key graph and pre-restricted to the current `key`. - **`self`** (and its Parts) is the write target. ## 1. `Diagram.trace(table_expr)` @@ -68,28 +69,28 @@ def trace(cls, table_expr: QueryExpression) -> Diagram: | Parameter | Type | Description | |---|---|---| -| `table_expr` | `QueryExpression` | A (possibly restricted) table expression. The trace propagates this restriction upstream through the FK graph. | +| `table_expr` | `QueryExpression` | The seed: one table with any restriction (see [Seeds](diagram.md#seeds)). | -Returns a `Diagram` instance whose nodes are the seed and all of its ancestors (across all loaded schemas), each carrying a restriction derived from the seed. +Returns an expanded `Diagram` in the `up` direction. Its nodes are the seed's table and every table reachable upward from it, across all loaded schemas, including the parts of every master reached. Each node holds that table's rows in `U(S)`, possibly none. ### Behavior -`trace` mirrors `cascade`: +1. Load the dependency graph via `connection.dependencies.load_all_upstream()`, the upstream analog of `load_all_downstream` ([#1423](https://github.com/datajoint/datajoint-python/issues/1423)). It discovers every schema reachable through foreign keys from the seed's schema, including schemas the current process has not activated. +2. Compute `U(S)` in one pass over the tables in reverse topological order: each table is restricted by its seed and by the restricted rows of its children; each master-part group by the masters its members reach, after which each part takes all part rows of those masters ([Computation](diagram.md#computation)). +3. Keep the seed's table and the tables reachable upward, with groups taken whole. Descendants of the seed and unrelated tables are not included. -1. Load the dependency graph via `connection.dependencies.load_all_upstream()` — the upstream analog of `load_all_downstream`, introduced with `Diagram.trace` ([#1423](https://github.com/datajoint/datajoint-python/issues/1423)). It discovers all schemas reachable via reverse FK edges from the seed's schema. -2. Take the seed's restriction and propagate it **upstream** along the FK graph. For each edge `parent → child`, when the child has a restriction, apply the upward rule (`U1`, `U2`, or `U3` per the cascade spec) to derive the parent's restriction. -3. Trim the resulting graph to **seed + ancestors only**. Descendants of the seed and unrelated ancestors are not included. -4. Convergence is **OR**: an ancestor entity is included if reachable through *any* FK path from the seed (consistent with how a child row "comes from" any of its FK parents). +The trace stays symbolic: no rows are read until a node is indexed, counted, or fetched. The one exception is a group where a foreign key between members does not map owning attributes onto owning attributes; its master keys are evaluated while the trace is built ([groups that iterate](diagram.md#groups-that-iterate)). -The graph traversal is dual to `cascade`'s. `cascade` walks `seed + descendants(seed)`; `trace` walks `seed + ancestors(seed)`. Both apply rules edge-by-edge; both terminate because the dependency graph is a DAG. +!!! version-changed "Changed in 2.4" + A trace includes all owned parts of every master it reaches, and the ancestors of those parts ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). In 2.3 it included a part only if the part lay on a foreign-key path to the seed. Upward crossings are always restrictions; 2.3 copied some conditions upward, which is correct only for keys of existing child rows. ### Indexing: `trace[T]` -The primary user-facing operation. Returns the ancestor's table pre-restricted through the FK join path. +The primary user-facing operation. Returns the table pre-restricted to its rows in the trace. ```python trace = dj.Diagram.trace(MyChild & key) -session = trace[Session] # QueryExpression: Session restricted to ancestors of MyChild & key +session = trace[Session] # QueryExpression: Session restricted to the rows MyChild & key depends on session.fetch1("session_date") trace[ExtractTraces].to_arrays("trace") ``` @@ -99,16 +100,18 @@ The argument may be: - **A Table subclass** (e.g. `Session`) — convenient when the class is in scope. - **A string** giving a table's fully-qualified or class name (e.g. `"experiment.Session"` or `"Session"`) — for when it isn't. -In both cases the result is the ancestor pre-restricted through the FK join path, returned as a `FreeTable` (a full `QueryExpression`, so every query operator and the fetch API apply). +In both cases the result is a full `QueryExpression`, so every query operator and the fetch API apply. -Requesting a table that is not an ancestor (or the seed itself) of `table_expr` raises `DataJointError`. This is the same guarantee that `cascade` provides for descendants. +Requesting a table that is not a node of the trace raises `DataJointError`. A node with zero rows, such as a part of an ancestor master that has no rows for this seed, returns an empty expression. ### Other methods | Method | Behavior | |---|---| -| `trace.counts()` | `dict[str, int]` mapping each contributing table's full table name (e.g. `` `imaging`.`subject` ``) to its row count under the seed's restriction. Includes the seed alongside its ancestors. Mirror of `cascade.counts()`. | -| `iter(trace)` | Iterate over pre-restricted `FreeTable`s in topological order (parents/roots first). | +| `trace.counts()` | `dict[str, int]` mapping each node's full table name (e.g. `` `imaging`.`subject` ``) to its row count. Includes the seed. Mirror of `cascade.counts()`. | +| `iter(trace)` | Iterate over pre-restricted `FreeTable`s in topological order (parents first). | +| `trace.prune()` | Remove nodes with zero rows. Allowed on traces, unlike cascades (new in 2.4). | +| `trace1 + trace2` | For two traces, the union of their rows, table by table (new in 2.4). | All `QueryExpression` operators (`join`, `aggr`, `proj`, `restrict`, `&`, `*`) work on the values returned by `trace[T]`. The trace is a *view* into the graph; the returned `QueryExpression`s are first-class. @@ -175,28 +178,47 @@ trace.counts() # '`imaging`.`__extract_traces`': 1, '`imaging`.`__summary`': 1} ``` -For a renamed-FK case (paralleling [Cascade Spec §Worked Example 1](cascade.md#example-1-part-of-part-with-renamed-fk)), the upward rules reverse the rename so `trace[Ancestor]` returns the ancestor with its native column names regardless of how the seed's columns are named. +For a renamed foreign key, the restriction maps the attributes back through `attr_map`, so `trace[Ancestor]` returns the ancestor with its own column names however the seed's columns are named ([S4](traversal-shapes.md#s4-renamed-foreign-key)). + +For a **merge table** — a master whose parts each reference a different upstream table — the trace from a downstream row reaches the merge master, its part row, and the upstream table that part names ([G5](traversal-shapes.md#g5-merge-table)): + +```python +trace = dj.Diagram.trace(Analysis & {"recording_id": 1}) +trace[Recording.Ephys] # the part row of recording 1 +trace[EphysSource] # the source that produced recording 1 +``` ## 2. `self.upstream` inside `make()` ### Where it's set -Before invoking `make(self, key)`, the framework records the current key but defers constructing the trace. `self.upstream = Diagram.trace(self & key)` is built the first time the user accesses `self.upstream` inside `make()` and memoized for the rest of the call; `make()` implementations that never read `self.upstream` never build the trace. Both the key and the memoized trace are cleared after `make()` returns (including when it raises). +Before invoking `make(self, key)`, the framework records the current key but defers constructing the trace. The trace is built the first time `make()` accesses `self.upstream`, and memoized for the rest of the call; `make()` implementations that never read `self.upstream` never build it. Both the key and the memoized trace are cleared after `make()` returns (including when it raises). + +The row for `key` does not exist yet while `make()` runs, so `self.upstream` cannot be a trace of `self & key`, which selects nothing. It is built from the parents that row will have: for each foreign key `e` in the table's primary key, the parent row `P_e & relabel_e(key)` (the key's values renamed onto `P_e`'s primary key), and -Once built, the trace diagram is reused for the remainder of the call. The underlying SQL is not issued until the user indexes into an ancestor: `self.upstream[Session]` builds a `QueryExpression`, and `.fetch1()` on that expression triggers a SELECT. Fetch results are not cached — each `self.upstream[T]` access returns a fresh `QueryExpression`, so reading the same ancestor twice issues two SELECTs. +``` +self.upstream = ⋃ over primary foreign keys e of Diagram.trace(P_e & relabel_e(key)) +``` + +the table-by-table union of those traces. Foreign keys outside the primary key are not included, because their values are not known before the insert. See [`self.upstream`](diagram.md#operations) in the Diagram specification. + +!!! version-changed "Changed in 2.4" + In 2.3, `self.upstream` was `Diagram.trace(self & key)`, which returned the parents only because the engine copied the key's condition upward. The 2.4 construction starts from the parent rows directly, and adds the parts of ancestor masters and their ancestors. + +Once built, the trace is reused for the remainder of the call. The underlying SQL is not issued until the user indexes into an ancestor: `self.upstream[Session]` builds a `QueryExpression`, and `.fetch1()` on that expression triggers a SELECT. Fetch results are not cached — each `self.upstream[T]` access returns a fresh `QueryExpression`, so reading the same ancestor twice issues two SELECTs. ### What it returns -`self.upstream[T]` returns the same value `dj.Diagram.trace(self & key)[T]` would return — the ancestor table pre-restricted to the rows that contributed to the current `key`. All `QueryExpression` operators work; the standard fetch API (`fetch`, `fetch1`, `to_arrays`, `to_dicts`, `to_pandas`) is fully supported. +`self.upstream[T]` returns the ancestor table `T` pre-restricted to the rows that the current `key` depends on. All `QueryExpression` operators work; the standard fetch API (`fetch`, `fetch1`, `to_arrays`, `to_dicts`, `to_pandas`) is fully supported. ### Allowed table set `self.upstream` exposes: -- All declared ancestors of `self` (transitively, including renamed-FK chains). -- The Parts of ancestors that themselves lie on an FK path to `self` — a Part is included only when it is genuinely reachable through the FK graph, not merely because its master is an ancestor. +- all ancestors of `self` reached through its primary foreign keys (transitively, including renamed foreign keys); +- **all owned parts of every ancestor master**, and their ancestors. By the group rule, a master-part group is one entity: once an ancestor master is in the trace, its whole group comes with it, not only those parts that happen to lie on a foreign-key path to `self`. This is the upstream mirror of `part_integrity="cascade"` treating the group as one. -Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. This is the same guarantee `Diagram.trace(...)` provides; `self.upstream` is just the per-`make()` instance of it. +Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. A table's own parts are descendants, not ancestors; read them directly as `self.PartName`. ### Examples @@ -286,6 +308,8 @@ Teams adopt the read surface incrementally: 2. **Start using `self.upstream` in new `make()` implementations.** Reads become ergonomic; nothing else changes. 3. **Migrate existing `make()` implementations** as opportunity allows. Replace `(Upstream & key).fetch(...)` with `self.upstream[Upstream].fetch(...)`. No semantic difference — but the read is now visibly confined to the declared upstream. +In 2.4, traces and `self.upstream` also include the parts of ancestor masters and their ancestors. Code that relied on `trace[T]` raising for such a table now receives its rows. + ## What is not in this specification - **Enforcement of the `make()` contract.** `trace` and `self.upstream` make the contract easy to follow and to inspect, but the open-source framework does not check it at runtime. Confirming that a `make()` reads only from its declared ancestors and writes only to its target, across every access path, is a code-inspection problem — performed by the DataJoint platform's code-deployment CI/CD (static analysis where sound, agentic review for the dynamic cases), backed by commit-pinned governed execution. See the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract). @@ -293,9 +317,10 @@ Teams adopt the read surface incrementally: ## References -- Source (shipped in 2.3): `src/datajoint/diagram.py` (`Diagram.trace`), `src/datajoint/autopopulate.py` (`AutoPopulate.upstream`). Implemented in [#1471](https://github.com/datajoint/datajoint-python/pull/1471) (trace) and [#1473](https://github.com/datajoint/datajoint-python/pull/1473) (self.upstream), building on the cascade rules from [#1468](https://github.com/datajoint/datajoint-python/pull/1468). -- Issues: [#1423](https://github.com/datajoint/datajoint-python/issues/1423) (Diagram.trace), [#1424](https://github.com/datajoint/datajoint-python/issues/1424) (self.upstream). -- [Cascade Specification](cascade.md) — propagation rules (F1/F2/F3 forward, U1/U2/U3 upward) shared with `trace`. +- Source: `src/datajoint/diagram.py` (`Diagram.expand`, `Diagram.trace`), `src/datajoint/autopopulate.py` (`AutoPopulate.upstream`). First implemented in [#1471](https://github.com/datajoint/datajoint-python/pull/1471) (trace) and [#1473](https://github.com/datajoint/datajoint-python/pull/1473) (self.upstream). +- Issues: [#1423](https://github.com/datajoint/datajoint-python/issues/1423) (Diagram.trace), [#1424](https://github.com/datajoint/datajoint-python/issues/1424) (self.upstream), [#1481](https://github.com/datajoint/datajoint-python/issues/1481) (parts of ancestor masters). +- [Diagram Specification](diagram.md#traversal-specification) — the definitions of `U` and the rules it is computed by. +- [Traversal Shape Catalog](traversal-shapes.md) — worked examples, with the rows each trace returns. +- [Cascade Specification](cascade.md) — the downstream case of the same traversal. - [AutoPopulate Specification](autopopulate.md) — `make()` execution model and the make() reproducibility contract. -- [Diagram Specification](diagram.md) — graph operations on the dependency graph. -- [Entity Integrity](../../explanation/entity-integrity.md) — schema dimensions and FK semantics. +- [Entity Integrity](../../explanation/entity-integrity.md) — schema dimensions and foreign-key semantics. diff --git a/src/reference/specs/traversal-shapes.md b/src/reference/specs/traversal-shapes.md new file mode 100644 index 00000000..71144be9 --- /dev/null +++ b/src/reference/specs/traversal-shapes.md @@ -0,0 +1,854 @@ +# Traversal Shape Catalog + +Worked examples of `Diagram.expand`, one per schema shape that the [Traversal specification](diagram.md#traversal-specification) distinguishes. + +!!! version-added "New in 2.4" + `Diagram.expand` and the behavior shown here are new in DataJoint 2.4. + +## How to read this catalog + +Each shape has a tiny schema, a few rows, one or more seeds, and the exact rows `expand` returns. The rows follow from the definitions of `D` and `U` in [Operations](diagram.md#operations); a sentence after each shape says why. The test suite mirrors these shapes one to one, under the same IDs. + +- **Directions.** *down* is `dj.Diagram.expand(seed, "down")`, which equals `Diagram.cascade(seed, part_integrity="cascade")`. *up* is `expand(seed, "up")`, which equals `Diagram.trace(seed)`. *both* is `expand(seed, "both")`. +- **Schemas.** Each shape is its own schema, declared with `schema = dj.Schema(...)`; table names repeat across shapes. Tables are `dj.Manual` unless the shape says otherwise. +- **Rows** are written as primary-key tuples, in the order of the primary key. When a column outside the primary key matters, it follows a semicolon: `(1, 2; NULL)` is a row with primary key `(1, 2)` whose foreign-key column is NULL. +- **Results** list every table in the diagram's node set (the [table-level reach](diagram.md#expanded-diagrams)). *none* means the table is a node with zero rows. + +| ID | Shape | What it pins down | +|---|---|---| +| [S1](#s1-identity-foreign-key-in-the-primary-key) | Identity foreign key in the primary key | The base case: the condition is copied | +| [S2](#s2-secondary-foreign-key) | Secondary foreign key | Upward crossing is a restriction | +| [S3](#s3-nullable-foreign-key-sharing-a-primary-key-column) | Nullable foreign key sharing a primary-key column | A foreign key with a NULL column references nothing | +| [S4](#s4-renamed-foreign-key) | Renamed foreign key | Crossing through `attr_map`, not by name | +| [S5](#s5-parallel-foreign-keys) | Parallel foreign keys | Separate edges combine with OR | +| [S6](#s6-diamond-with-a-shared-key) | Diamond with a shared key | Paths combine by union | +| [S7](#s7-diamond-reached-through-a-secondary-foreign-key) | Diamond reached through a secondary foreign key | A row reached only through its secondary reference | +| [G1](#g1-part-to-master-lift) | Part-to-master lift | The lift is existential | +| [G2](#g2-part-of-part-with-two-chains) | Part-of-part with two chains | The lift is one step on the master key | +| [G3](#g3-a-second-renamed-foreign-key-to-its-own-master) | A second, renamed foreign key to its own master | Only the owning reference lifts | +| [G4](#g4-renamed-sibling-foreign-key-chained-across-three-instances) | Renamed sibling foreign key, chained across three instances | The group step iterates | +| [G5](#g5-merge-table) | Merge table | A trace reaches the parts of an ancestor master | +| [G6](#g6-a-part-its-master-does-not-own) | A part its master does not own | Traversed as a plain table | +| [G7](#g7-owned-part-of-part-with-renamed-attributes) | Owned part-of-part with renamed attributes | The lift renames the owning attributes | +| [K1](#k1-seed-restricted-by-a-descendant) | Seed restricted by a descendant | Seeds are materialized before deleting | +| [K2](#k2-self-referencing-seed) | Self-referencing seed | Same, for a seed that reads its own table | +| [K3](#k3-seed-kinds-give-the-same-rows) | Seed kinds give the same rows | Dict, list, string, subquery, `AndList`, `Not` | +| [K4](#k4-djtop-seed) | `dj.Top` seed | `Top` is replaced by its keys | +| [K5](#k5-empty-seeds) | Empty seeds | Nothing in, nothing out | +| [K6](#k6-unrestricted-seed) | Unrestricted seed | All rows, and a warning for a dict that names no attribute | +| [X1](#x1-unloaded-downstream-schema) | Unloaded downstream schema | Schemas are discovered through foreign keys | +| [X2](#x2-group-cycle-refused-at-declaration) | Group cycle refused at declaration | The group quotient stays acyclic | + +--- + +## Foreign-key shapes + +### S1. Identity foreign key in the primary key + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Session` | `subject_id, session_id` | (1, 1), (1, 2), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & {"subject_id": 1}` | down | `Subject`: 1
`Session`: (1, 1), (1, 2) | +| `Session & {"session_id": 1}` | up | `Session`: (1, 1), (2, 1)
`Subject`: 1, 2 | + +**Why.** The foreign key embeds the parent's whole key under the same names, in the child's primary key, so the condition `subject_id = 1` is copied to `Session` unchanged. Upward, `Subject` is restricted by the restricted sessions, which reference subjects 1 and 2. + +### S2. Secondary foreign key + +```python +@schema +class Rig(dj.Lookup): + definition = """ + rig_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + --- + -> Rig + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Rig` | `rig_id` | 1, 2, 3 | +| `Session` | `session_id; rig_id` | (1; 1), (2; 1), (3; 2) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Rig & {"rig_id": 1}` | down | `Rig`: 1
`Session`: 1, 2 | +| `Session & {"session_id": 3}` | up | `Session`: 3
`Rig`: 2 | + +**Why.** Downward, the condition names only `rig_id`, the foreign key's own non-null attribute, so it is copied. Upward, the seed's condition names `session_id`, which the foreign key does not carry: `Rig` is restricted by the restricted session's `rig_id`, giving rig 2. Relabeling the dict would drop `session_id` and select every rig. + +### S3. Nullable foreign key sharing a primary-key column + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +@schema +class Surgery(dj.Manual): + definition = """ + -> Subject + surgery_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + --- + -> [nullable] Surgery + """ +``` + +`Session`'s foreign key to `Surgery` is `(subject_id, surgery_id)`. It shares `subject_id` with `Session`'s primary key, so only `surgery_id` can be NULL. + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Surgery` | `subject_id, surgery_id` | (1, 1), (1, 2), (2, 1) | +| `Session` | `subject_id, session_id; surgery_id` | (1, 1; 1), (1, 2; NULL), (1, 3; 2), (2, 1; 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Surgery & {"subject_id": 1}` | down | `Surgery`: (1, 1), (1, 2)
`Session`: (1, 1), (1, 3) | +| `Session & {"subject_id": 1, "session_id": 2}` | up | `Session`: (1, 2)
`Subject`: 1
`Surgery`: none | + +**Why.** Session (1, 2) has `subject_id` set and `surgery_id` NULL, so it references no surgery (A2). Downward, the foreign key has a nullable column, so `subject_id = 1` is not copied; a copy would also delete session (1, 2) ([#1590](https://github.com/datajoint/datajoint-python/issues/1590)). Upward, session (1, 2) reaches its subject but no surgery. + +### S4. Renamed foreign key + +```python +@schema +class Neuron(dj.Manual): + definition = """ + neuron_id : int32 + """ + + +@schema +class Input(dj.Manual): + definition = """ + -> Neuron.proj(source_id="neuron_id") + input_id : int32 + --- + neuron_id : int32 # a local label, not a foreign key + """ +``` + +`Input` holds the referenced neuron as `source_id`, and also has an unrelated attribute that happens to be named `neuron_id`. + +| Table | Columns | Rows | +|---|---|---| +| `Neuron` | `neuron_id` | 1, 2, 3 | +| `Input` | `source_id, input_id; neuron_id` | (1, 1; 2), (1, 2; 3), (2, 1; 1), (3, 1; 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Neuron & {"neuron_id": 1}` | down | `Neuron`: 1
`Input`: (1, 1), (1, 2) | +| `Input & {"source_id": 2, "input_id": 1}` | up | `Input`: (2, 1)
`Neuron`: 2 | + +**Why.** A renamed foreign key is crossed through its `attr_map` (`source_id` ↔ `neuron_id`), never by attribute name. Copying `neuron_id = 1` to `Input` would match the unrelated label and select inputs (2, 1) and (3, 1). + +### S5. Parallel foreign keys + +```python +@schema +class Neuron(dj.Manual): + definition = """ + neuron_id : int32 + """ + + +@schema +class Synapse(dj.Manual): + definition = """ + -> Neuron + -> Neuron.proj(target_id="neuron_id") + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Neuron` | `neuron_id` | 1, 2, 3 | +| `Synapse` | `neuron_id, target_id` | (1, 2), (2, 3), (3, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Neuron & {"neuron_id": 1}` | down | `Neuron`: 1
`Synapse`: (1, 2), (3, 1) | +| `Synapse & {"neuron_id": 1, "target_id": 2}` | up | `Synapse`: (1, 2)
`Neuron`: 1, 2 | +| `Neuron & {"neuron_id": 1}` | both | `Neuron`: 1, 2, 3
`Synapse`: (1, 2), (3, 1) | + +**Why.** The two foreign keys are separate edges, and a synapse is reached if either one references neuron 1; combining them with AND would select only synapses from neuron 1 to itself, of which there are none. *both* adds neurons 2 and 3, the other ends of the exported synapses; `D(S) ∪ U(S)` would leave them out. + +### S6. Diamond with a shared key + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + """ + + +@schema +class Implant(dj.Manual): + definition = """ + -> Subject + implant_id : int32 + """ + + +@schema +class Recording(dj.Manual): + definition = """ + -> Session + -> Implant + """ +``` + +`Recording` inherits `subject_id` through both parents, as one shared column. + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Session` | `subject_id, session_id` | (1, 1), (1, 2), (2, 1) | +| `Implant` | `subject_id, implant_id` | (1, 1), (1, 2), (2, 1) | +| `Recording` | `subject_id, session_id, implant_id` | (1, 1, 1), (1, 2, 2), (2, 1, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & {"subject_id": 1}` | down | `Subject`: 1
`Session`: (1, 1), (1, 2)
`Implant`: (1, 1), (1, 2)
`Recording`: (1, 1, 1), (1, 2, 2) | +| `Implant & {"subject_id": 1, "implant_id": 1}` | down | `Implant`: (1, 1)
`Recording`: (1, 1, 1) | +| `Implant & {"subject_id": 1, "implant_id": 1}` | both | `Implant`: (1, 1)
`Recording`: (1, 1, 1)
`Subject`: 1
`Session`: (1, 1) | +| `Recording & {"subject_id": 1, "session_id": 2, "implant_id": 2}` | up | `Recording`: (1, 2, 2)
`Session`: (1, 2)
`Implant`: (1, 2)
`Subject`: 1 | + +**Why.** From `Subject`, both paths carry the same `subject_id`, and their union is the result; nothing is intersected. *both* from the implant adds session (1, 1), the other parent of the exported recording, and the subject. + +### S7. Diamond reached through a secondary foreign key + +```python +@schema +class Lab(dj.Manual): + definition = """ + lab_id : int32 + """ + + +@schema +class Rig(dj.Manual): + definition = """ + rig_id : int32 + --- + -> Lab + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Lab + session_id : int32 + --- + -> Rig + """ +``` + +A session belongs to a lab through its primary key and uses a rig, which may belong to another lab, through a secondary foreign key. `Session`'s own `lab_id` and its rig's `lab_id` are independent. + +| Table | Columns | Rows | +|---|---|---| +| `Lab` | `lab_id` | 1, 2 | +| `Rig` | `rig_id; lab_id` | (10; 1), (20; 2) | +| `Session` | `lab_id, session_id; rig_id` | (1, 1; 20), (2, 1; 10), (2, 2; 20) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Lab & {"lab_id": 1}` | down | `Lab`: 1
`Rig`: 10
`Session`: (1, 1), (2, 1) | +| `Session & {"lab_id": 2, "session_id": 1}` | up | `Session`: (2, 1)
`Rig`: 10
`Lab`: 1, 2 | +| `Rig & {"rig_id": 10}` | both | `Rig`: 10
`Session`: (2, 1)
`Lab`: 1, 2 | + +**Why.** Session (2, 1) belongs to lab 2 but uses rig 10 of lab 1, so it reaches the seed only through `Rig`. `Rig`'s condition, `lab_id = 1`, names an attribute that `Session`'s foreign key to `Rig` does not carry, so it is not copied; `Session` is restricted by the restricted rigs' `rig_id`. Copying it would match sessions on their own `lab_id` and miss session (2, 1), and the DELETE on `Rig` would then fail on its foreign key. + +--- + +## Group shapes + +### G1. Part-to-master lift + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Trial(dj.Part): + definition = """ + -> master + trial_id : int32 + """ + + +@schema +class Review(dj.Manual): + definition = """ + -> Session + review_id : int32 + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Session` | `session_id` | 1, 2, 3 | +| `Session.Trial` | `session_id, trial_id` | (1, 1), (1, 2), (2, 1), (3, 1) | +| `Review` | `session_id, review_id` | (1, 1), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session.Trial & {"trial_id": 2}` | down | `Session.Trial`: (1, 1), (1, 2)
`Session`: 1
`Review`: (1, 1) | +| `Session.Trial & {"trial_id": 2}` | up | `Session.Trial`: (1, 1), (1, 2)
`Session`: 1 | + +**Why.** The lift selects the sessions that have a matching trial, `Session & (Session.Trial & {"trial_id": 2})`: only session 1. Sessions 2 and 3, and review (2, 1), are untouched. The old upward copy of `trial_id = 2` named no attribute of `Session` and selected every session ([#1591](https://github.com/datajoint/datajoint-python/issues/1591)). Upward, the instance is read whole, so trial (1, 1) comes with trial (1, 2). + +The same seed under the other delete modes: + +| `part_integrity` | Result | +|---|---| +| `"cascade"` | Deletes the *down* rows above | +| `"ignore"` | Deletes trial (1, 2) only | +| `"enforce"` | Raises before deleting anything: trial (1, 2) would be deleted without session 1 | + +### G2. Part-of-part with two chains + +```python +@schema +class Ext(dj.Manual): + definition = """ + ext_id : int32 + """ + + +@schema +class Master(dj.Manual): + definition = """ + master_id : int32 + """ + + class Q(dj.Part): + definition = """ + -> master + q_id : int32 + --- + -> Ext + """ + + class P(dj.Part): + definition = """ + -> master + p_id : int32 + --- + -> Master.Q + -> Ext + """ +``` + +`Master.P` is reachable from `Master` through two chains: directly, and through `Master.Q`. Each part also references `Ext`. + +| Table | Columns | Rows | +|---|---|---| +| `Ext` | `ext_id` | 1, 2 | +| `Master` | `master_id` | 1, 2, 3 | +| `Master.Q` | `master_id, q_id; ext_id` | (1, 10; 1), (2, 20; 2), (3, 30; 2) | +| `Master.P` | `master_id, p_id; q_id, ext_id` | (1, 100; 10, 2), (2, 200; 20, 1), (3, 300; 30, 2) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Ext & {"ext_id": 1}` | down | `Ext`: 1
`Master.Q`: (1, 10), (2, 20)
`Master.P`: (1, 100), (2, 200)
`Master`: 1, 2 | +| `Master.P & {"master_id": 1, "p_id": 100}` | up | `Master.P`: (1, 100)
`Master`: 1
`Master.Q`: (1, 10)
`Ext`: 1, 2 | + +**Why.** Downward, `Ext` 1 reaches Q (1, 10) and P (2, 200), which lift to masters 1 and 2; each master brings its whole instance, and master 3 is untouched. The lift is one step on `master_id`, which every part row carries, so the second chain needs no separate walk. Upward, the instance of P (1, 100) includes Q (1, 10), and the two parts reference `Ext` 1 and 2. + +### G3. A second, renamed foreign key to its own master + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Link(dj.Part): + definition = """ + -> master + link_id : int32 + --- + -> master.proj(other_session="session_id") + """ +``` + +`Session.Link` is owned through `-> master`, so its owning attribute is `session_id`. It points to another session through a second, renamed foreign key. + +| Table | Columns | Rows | +|---|---|---| +| `Session` | `session_id` | 1, 2, 3 | +| `Session.Link` | `session_id, link_id; other_session` | (1, 1; 2), (3, 1; 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session.Link & {"session_id": 1}` | down | `Session.Link`: (1, 1), (3, 1)
`Session`: 1, 3 | +| `Session.Link & {"session_id": 1}` | up | `Session.Link`: (1, 1)
`Session`: 1, 2 | + +**Why.** Link (1, 1) lifts through its owning attribute to session 1, not to session 2. Deleting session 1 then deletes link (3, 1), which points to it, and so the rest of instance 3. Upward, link (1, 1) reads session 2, a parent, but not link (3, 1), a child. The second foreign key maps `other_session`, which is not an owning attribute, onto the master's key; it links one instance to another, so this group [iterates](diagram.md#groups-that-iterate). + +With `part_integrity="enforce"`, `(Session & {"session_id": 1}).delete()` raises: it would delete link (3, 1) without session 3. + +### G4. Renamed sibling foreign key, chained across three instances + +```python +@schema +class Run(dj.Manual): + definition = """ + run_id : int32 + """ + + class Step(dj.Part): + definition = """ + -> master + step_id : int32 + """ + + class Carry(dj.Part): + definition = """ + -> master + carry_id : int32 + --- + -> Run.Step.proj(prev_run="run_id", prev_step="step_id") + """ +``` + +A `Carry` row of one run references a `Step` row of an earlier run. + +| Table | Columns | Rows | +|---|---|---| +| `Run` | `run_id` | 1, 2, 3, 4 | +| `Run.Step` | `run_id, step_id` | (1, 1), (2, 1), (3, 1), (4, 1) | +| `Run.Carry` | `run_id, carry_id; prev_run, prev_step` | (2, 1; 1, 1), (3, 1; 2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Run & {"run_id": 1}` | down | `Run`: 1, 2, 3
`Run.Step`: (1, 1), (2, 1), (3, 1)
`Run.Carry`: (2, 1), (3, 1) | +| `Run & {"run_id": 2}` | down | `Run`: 2, 3
`Run.Step`: (2, 1), (3, 1)
`Run.Carry`: (2, 1), (3, 1) | +| `Run & {"run_id": 2}` | up | `Run`: 1, 2
`Run.Step`: (1, 1), (2, 1)
`Run.Carry`: (2, 1) | +| `Run & {"run_id": 2}` | both | `Run`: 1, 2, 3
`Run.Step`: (1, 1), (2, 1), (3, 1)
`Run.Carry`: (2, 1), (3, 1) | + +**Why.** Deleting run 1 deletes step (1, 1), so carry (2, 1), which references it, and so the rest of run 2; that reaches carry (3, 1) and run 3. Run 4 is untouched. One pass over the group would stop at run 1. `Carry`'s reference to `Step` maps `prev_run`, not `Carry`'s owning attribute `run_id`, onto `Step`'s owning attribute, so the engine repeats the group step: the master keys grow as {1}, {1, 2}, {1, 2, 3}, and a fourth round adds nothing. Upward the chain runs the other way, from run 2 back to run 1. + +### G5. Merge table + +```python +@schema +class EphysSource(dj.Manual): + definition = """ + ephys_id : int32 + """ + + +@schema +class ImagingSource(dj.Manual): + definition = """ + imaging_id : int32 + """ + + +@schema +class Recording(dj.Manual): + definition = """ + recording_id : int32 + """ + + class Ephys(dj.Part): + definition = """ + -> master + --- + -> EphysSource + """ + + class Imaging(dj.Part): + definition = """ + -> master + --- + -> ImagingSource + """ + + +@schema +class Analysis(dj.Manual): + definition = """ + -> Recording + """ +``` + +`Recording` merges two upstream sources: each recording has one part row, which names its source. + +| Table | Columns | Rows | +|---|---|---| +| `EphysSource` | `ephys_id` | 1, 2 | +| `ImagingSource` | `imaging_id` | 1 | +| `Recording` | `recording_id` | 1, 2, 3 | +| `Recording.Ephys` | `recording_id; ephys_id` | (1; 1), (2; 2) | +| `Recording.Imaging` | `recording_id; imaging_id` | (3; 1) | +| `Analysis` | `recording_id` | 1, 2, 3 | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Analysis & {"recording_id": 1}` | up | `Analysis`: 1
`Recording`: 1
`Recording.Ephys`: 1
`Recording.Imaging`: none
`EphysSource`: 1
`ImagingSource`: none | +| `EphysSource & {"ephys_id": 2}` | down | `EphysSource`: 2
`Recording.Ephys`: 2
`Recording`: 2
`Recording.Imaging`: none
`Analysis`: 2 | + +```python +trace = dj.Diagram.trace(Analysis & {"recording_id": 1}) +trace[EphysSource] # ephys source 1 +trace[ImagingSource] # empty: a node with zero rows, not an error +``` + +**Why.** Upward, recording 1 brings its whole instance. Its `Ephys` part row references the source that produced the recording, so `trace[EphysSource]` resolves ([#1481](https://github.com/datajoint/datajoint-python/issues/1481)). Downward, deleting ephys source 2 deletes the part row that names it, so the whole of recording 2, and its analysis. With `part_integrity="enforce"` that delete raises, because the part row would go without its master. + +### G6. A part its master does not own + +```python +@schema +class Probe(dj.Manual): + definition = """ + probe_id : int32 + """ + + class Channel(dj.Part): + definition = """ + channel_id : int32 + --- + -> master + """ +``` + +`Probe.Channel` references its master only through a secondary foreign key, so the master does not own it. Declaring it raises a warning, and it is traversed as a plain table: no lift and no instance. A nullable reference, `-> [nullable] master`, has the same effect. + +| Table | Columns | Rows | +|---|---|---| +| `Probe` | `probe_id` | 1, 2 | +| `Probe.Channel` | `channel_id; probe_id` | (1; 1), (2; 1), (3; 2) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Probe.Channel & {"channel_id": 1}` | down | `Probe.Channel`: 1 | +| `Probe.Channel & {"channel_id": 1}` | up | `Probe.Channel`: 1
`Probe`: 1 | +| `Probe & {"probe_id": 1}` | down | `Probe`: 1
`Probe.Channel`: 1, 2 | + +**Why.** As a plain table, the channel reaches its probe only as a parent: downward nothing lifts, and upward channel 2 is not part of channel 1's instance. Deleting the probe still deletes its channels, as children. Moving the reference into the primary key, as `-> master` or renamed as `-> master.proj(host_id="probe_id")`, would make the part owned. + +### G7. Owned part-of-part with renamed attributes + +The part-of-part shape of [#1429](https://github.com/datajoint/datajoint-python/issues/1429), Case 1: + +```python +@schema +class Master(dj.Manual): + definition = """ + master_id : int32 + """ + + class PartA(dj.Part): + definition = """ + -> master + part_a_id : int32 + """ + + class PartB(dj.Part): + definition = """ + -> Master.PartA.proj(src_master="master_id", src_part="part_a_id") + part_b_id : int32 + """ + + +@schema +class Report(dj.Manual): + definition = """ + -> Master + report_id : int32 + """ +``` + +`PartB` has no attribute named `master_id`. It is owned through its primary-key reference to `PartA`, which is owned, and its owning attribute is `src_master`, which that reference maps onto `PartA`'s `master_id`. + +| Table | Columns | Rows | +|---|---|---| +| `Master` | `master_id` | 1, 2, 3 | +| `Master.PartA` | `master_id, part_a_id` | (1, 10), (1, 11), (2, 20) | +| `Master.PartB` | `src_master, src_part, part_b_id` | (1, 10, 100), (1, 11, 110), (2, 20, 200) | +| `Report` | `master_id, report_id` | (1, 1), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Master.PartB & {"part_b_id": 100}` | down | `Master.PartB`: (1, 10, 100), (1, 11, 110)
`Master`: 1
`Master.PartA`: (1, 10), (1, 11)
`Report`: (1, 1) | +| `Master.PartB & {"part_b_id": 100}` | up | `Master.PartB`: (1, 10, 100), (1, 11, 110)
`Master`: 1
`Master.PartA`: (1, 10), (1, 11) | + +**Why.** The lift renames the owning attribute onto the master's key, `Master & R(PartB).proj(master_id="src_master")`, and selects master 1 in one step. Master 1 brings its whole instance, including part B row (1, 11, 110), which the seed did not name, and, downward, its report. The reference from `PartB` to `PartA` maps `src_master` onto `PartA`'s owning attribute, so the group needs no iteration. With `part_integrity="enforce"`, deleting the seed raises, because part B row (1, 10, 100) would go without master 1. + +--- + +## Seed shapes + +K1 to K6 share one schema: + +```python +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + --- + species : varchar(16) + """ + + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_id : int32 + --- + quality : int32 + """ + + +@schema +class Trial(dj.Manual): + definition = """ + -> Session + trial_id : int32 + """ + + +@schema +class Flag(dj.Manual): + definition = """ + flag_id : int32 + --- + -> [nullable] Subject + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id; species` | (1; mouse), (2; rat), (3; mouse), (4; rat) | +| `Session` | `subject_id, session_id; quality` | (1, 1; 0), (1, 2; 1), (2, 1; 1), (2, 2; 0), (3, 1; 1) | +| `Trial` | `subject_id, session_id, trial_id` | (1, 1, 1), (1, 2, 1), (2, 1, 1), (2, 2, 1), (3, 1, 1) | +| `Flag` | `flag_id; subject_id` | (1; 1), (2; 3) | + +Subject 4 has no sessions. + +### K1. Seed restricted by a descendant + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & (Session & "quality = 0")` | down | `Subject`: 1, 2
`Session`: (1, 1), (1, 2), (2, 1), (2, 2)
`Trial`: (1, 1, 1), (1, 2, 1), (2, 1, 1), (2, 2, 1)
`Flag`: 1 | +| `Subject & (Session & "quality = 0")` | up | `Subject`: 1, 2 | + +**Why.** The seed selects subjects 1 and 2, each with a session of quality 0, and *down* takes all their sessions, not only those. The seed's condition reads `Session`, which the delete empties before `Subject`, so the seed is materialized to the keys of subjects 1 and 2 before the first DELETE. Evaluated live, the DELETE on `Subject` would match nothing, and the subjects would remain ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)). + +### K2. Self-referencing seed + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session & (dj.U("subject_id") & (Session & "quality = 0"))` | down | `Session`: (1, 1), (1, 2), (2, 1), (2, 2)
`Trial`: (1, 1, 1), (1, 2, 1), (2, 1, 1), (2, 2, 1) | +| `Session & (dj.U("subject_id") & (Session & "quality = 0"))` | up | `Session`: (1, 1), (1, 2), (2, 1), (2, 2)
`Subject`: 1, 2 | + +**Why.** The seed selects every session of a subject that has a session of quality 0, and its condition reads `Session` itself, which is deleted no later than the seed. It is materialized before the first DELETE; this also avoids MySQL's refusal of a DELETE whose subquery reads the table being deleted (error 1093). + +### K3. Seed kinds give the same rows + +Each seed below selects sessions (2, 1) and (2, 2), the sessions of subject 2: + +| Kind | Seed | +|---|---| +| dict | `Session & {"subject_id": 2}` | +| list of dicts | `Session & [{"subject_id": 2, "session_id": 1}, {"subject_id": 2, "session_id": 2}]` | +| string | `Session & "subject_id = 2"` | +| string with a subquery | `Session & f"subject_id IN (SELECT subject_id FROM {Subject.full_table_name} WHERE species = 'rat')"` | +| query expression | `Session & (Subject & {"species": "rat"})` | +| `AndList` | `Session & dj.AndList([{"subject_id": 2}, "quality >= 0"])` | +| `Not` | `Session & dj.Not(Flag)`, or `Session - Flag` | + +| Seed | Direction | Rows returned | +|---|---|---| +| any of the above | down | `Session`: (2, 1), (2, 2)
`Trial`: (2, 1, 1), (2, 2, 1) | +| any of the above | up | `Session`: (2, 1), (2, 2)
`Subject`: 2 | +| `Session & dj.Not(Flag)` after inserting `Flag` (3; NULL) | down | `Session`: none
`Trial`: none | +| `Session & dj.Not(Flag)` after inserting `Flag` (3; NULL) | up | `Session`: none
`Subject`: none | + +**Why.** Literal keys cross edges by relabeling and the other kinds cross by restriction; both give the same rows. Upward, only subject 2 is reached, though subject 4 is also a rat, because no selected session references it. The last two rows are the NOT IN NULL trap: `Session & dj.Not(Flag)` compiles to `subject_id NOT IN (SELECT subject_id FROM flag)`, and once that list holds a NULL, SQL's `NOT IN` is never true. The seed then selects no rows, and so does its expansion, in every direction; the materialized form `(Session & dj.Not(Flag)).keys()` is empty too. `expand` takes the rows the database selects for the seed and does not reinterpret its condition. + +### K4. `dj.Top` seed + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session & "quality = 1" & dj.Top(2)` | down | `Session`: (1, 2), (2, 1)
`Trial`: (1, 2, 1), (2, 1, 1) | +| `Session & "quality = 1" & dj.Top(2)` | up | `Session`: (1, 2), (2, 1)
`Subject`: 1, 2 | + +**Why.** Three sessions have quality 1; `dj.Top(2)` keeps the first two in primary-key order. The `Top` is replaced by those two keys before traversal. Ignoring it would delete session (3, 1) as well ([#1589](https://github.com/datajoint/datajoint-python/issues/1589)). + +### K5. Empty seeds + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session & False`, `Session & []`, or `Session & {"subject_id": 4}` | down | `Session`: none
`Trial`: none | +| the same | up | `Session`: none
`Subject`: none | + +**Why.** Each seed selects no rows, so each expansion is empty, and the nodes are still present. Subject 4 exists, but no selected session references it: a key dict crosses upward by restriction, not by relabeling, so `{"subject_id": 4}` does not reach it. + +### K6. Unrestricted seed + +| Seed | Direction | Rows returned | +|---|---|---| +| `Session` | down | `Session`: all five rows
`Trial`: all five rows | +| `Session` | up | `Session`: all five rows
`Subject`: 1, 2, 3 | +| `Session & {"species": "rat"}` | down, up | The same rows as `Session`, and a warning | + +**Why.** An unrestricted seed selects every row. Upward, subject 4 is not reached, because no session references it. `species` is not an attribute of `Session`, so the dict means all rows, as in any restriction. As a seed it raises a warning, because a misspelled attribute would otherwise select every row. + +--- + +## Graph shapes + +### X1. Unloaded downstream schema + +```python +# module lab.py; the only schema this process activates +lab = dj.Schema("shapes_lab") + + +@lab +class Subject(dj.Manual): + definition = """ + subject_id : int32 + """ + + +# module analysis.py; declared earlier, never imported by this process +analysis = dj.Schema("shapes_analysis") + + +@analysis +class Score(dj.Manual): + definition = """ + -> Subject + score_id : int32 + """ +``` + +| Table | Columns | Rows | +|---|---|---| +| `Subject` | `subject_id` | 1, 2 | +| `Score` (in `shapes_analysis`) | `subject_id, score_id` | (1, 1), (2, 1) | + +| Seed | Direction | Rows returned | +|---|---|---| +| `Subject & {"subject_id": 1}` | down | `Subject`: 1
`Score`: (1, 1) | + +**Why.** `expand` first loads every schema that references a loaded schema through foreign keys, so `shapes_analysis` joins the graph even though none of its classes is in scope. Index its table by its full table name. + +### X2. Group cycle refused at declaration + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Trial(dj.Part): + definition = """ + -> master + trial_id : int32 + """ + + +@schema +class Review(dj.Manual): + definition = """ + -> Session + """ +``` + +Later, a new part is added to `Session` that references `Review`: + +```python +@schema +class Session(dj.Manual): + definition = """ + session_id : int32 + """ + + class Trial(dj.Part): + definition = """ + -> master + trial_id : int32 + """ + + class Note(dj.Part): + definition = """ + -> master + --- + -> Review + """ +``` + +**Result.** Declaring `Session.Note` raises `DataJointError` before any table is created. The message names the part (`Session.Note`), its parent (`Review`), and the cycle: `Session` → `Review` → `Session.Note`, which belongs to `Session`'s group. No rows change. + +**Why.** `Session.Note` is owned, because `-> master` is in its primary key. An owned part may reference its master, an owned part of the same master, or a table that does not depend on its master, with groups taken whole; parts the master does not own count as outside the group ([#1619](https://github.com/datajoint/datajoint-python/issues/1619)). `Review` depends on `Session`, so `Session.Note` would make the group its own ancestor: a cycle in the group quotient `Q`, and DataJoint could no longer order these tables for delete, drop, or trace (T8). Two groups whose owned parts reference each other's masters form the same kind of cycle and are refused the same way. A part that is not owned is a plain table, so its references cannot close a group cycle. + +--- + +## See also + +- [Diagram Specification](diagram.md#traversal-specification) — the definitions these examples follow +- [Cascade Specification](cascade.md) — deletes +- [Upstream Trace Specification](trace.md) — `trace` and `self.upstream` +- [Master-Part Specification](master-part.md) — master-part groups