fix(declare): refuse a part reference that closes a master-part group cycle

[dimitri-yatsenko]

Closes #1619.

Owned parts

This PR decides which parts belong to their master by a rule instead of by the table name.

A part is owned by its master when its primary key includes a reference to the master, either directly or through another part the master owns. The reference may rename the key attributes. For example:

• -> master in the primary key: owned.
• -> master.proj(session="session_id") in the primary key: owned.
• -> master.Channel in the primary key, where Channel is owned: owned. This includes renamed references, such as the part-of-part case in Cascade may fail when Part table references another Part with renamed foreign keys #1429.
• -> master or -> [nullable] master below the --- line, or no reference to the master at all: not owned.
• A primary-key reference only to a part that is not owned: not owned.

An owned part and its master form one group: a master row and its owned part rows are one entity. A part that is not owned is ordered as a separate table, like any other table that references the master.

What changes

Declaration refuses a group cycle. Part._declare_check runs after the definition is parsed and before any DDL. It finds out whether the new part is owned. For an owned part, it checks each reference to a table outside the group. If that table already depends on the master, the declaration is refused. Parts the master does not own count as outside the group. For the reproduction in the issue:

DataJointError: Part table `lab`.`session__summary` cannot reference `lab`.`analysis`, which depends on
the part's master `lab`.`session`. The reference would close the cycle `lab`.`session` -> `lab`.`analysis`
-> `lab`.`session__summary` -> `lab`.`session` (each table references the one before it or is the master of
the part before it), and DataJoint could no longer order these tables for delete, drop or trace. A part
table may reference its master, a part that its master owns, or a table that does not depend on its master.

When the referenced table is a part that the master does not own, the message says so and explains why.

topo_sort uses owned parts. Only owned parts are kept right after their masters, and a master is ordered after the tables its owned parts reference. Other parts are sorted as separate tables. An existing cycle now raises a DataJointError that prints the cycle and names the part that closes it. It also says how to recover: drop that part with drop(part_integrity='ignore') and declare it without the reference. Before, the error was networkx.exception.NetworkXUnfeasible.

Only the needed tables are sorted. topo_sort(graph, nodes) takes an optional set of tables. It sorts those tables together with the tables on dependency paths between them, so their order still respects every dependency between them, and it returns only the requested tables. Ownership is judged on the whole graph. The callers now pass their own tables:

• the cascade and trace propagation in Diagram: the seed's descendants or ancestors. With part_integrity="cascade", the cascade also includes the masters that its current engine reaches from their parts.
• Diagram.__iter__ / __reversed__, used by delete, drop and counts(): nodes_to_show.
• Schema.list_tables(): the schema's own tables.

With an existing cycle, only the operations that must order it now fail. Operations on other tables keep working, and so does dropping the part that closes the cycle.

Warning for a part that is not owned. It is raised at declaration and says that DataJoint sorts the part as a separate table, and that from DataJoint 2.4 delete and trace will also treat it as one. It also says how to make the part owned.

Supporting changes

• owned_parts(graph) reads ownership from the dependency graph. Each foreign key is its own edge, renamed ones included, with a primary flag that is set when all its attributes are in the referencing table's primary key. A part is owned when a primary edge comes from its master or from an owned part of the same master.
• Other new helpers in dependencies.py:
  • is_owned handles a part that is being declared.
  • group_graph is the dependency graph with an edge from each owned part to its master.
  • group_ancestors and group_cycle work on that graph.
• declare(), prepare_declare() and compile_foreign_key() take an optional foreign_keys list, filled with one (parent_table, attr_map) entry per foreign key. fk_attribute_map could not be used here: it keeps one parent per attribute, so a reference whose attributes are all shared with a later reference is missing from it. A new part is owned under the same rule as the graph uses: some reference to the master or to an owned part has all its attributes in the part's primary key.
• The private _declare_check hook receives foreign_keys as a third argument. The AutoPopulate override is updated to match.
• Dependencies.load_all_upstream() takes optional extra schemas to start from. The check passes the schemas of the referenced tables.

Why the check is exact

With owned parts as groups and every other table on its own, topo_sort succeeds exactly when the groups do not depend on each other in a cycle. A new table has no children, so it cannot be on a cycle itself. A plain table, a master, or a part that is not owned adds no ordering constraint beyond its own references, so it can never close a cycle. That includes a master declared after a part of its own that does not reference it.

An owned part adds one constraint: its master must come after each table outside the group that the part references. A cycle forms exactly when such a table X already depends on the master. "Depends on" follows references upward and, at a master, continues through the references of its owned parts. group_cycle tests exactly this.

Changes for existing schemas

• Newly refused, and newly failing in topo_sort: an owned part that references a part its master does not own, when that part depends on the master. An example is a part with -> [nullable] master or a secondary -> master. Name-based grouping sorted such schemas, because the sibling reference stayed inside the group. Under the new rule the sibling is a separate table that depends on the master, so the owned part closes a cycle. Existing schemas of this shape now get the cycle error, with recovery instructions, for the operations that order those tables. This is the only shape that used to sort and no longer does.
• Newly accepted: a part that is not owned may now reference tables that depend on its master. For example, a part with no reference to its master that references a table downstream of the master. Name-based grouping failed on these.
• Order: parts that are not owned are no longer moved next to their masters in topo_sort results, including list_tables() and Diagram iteration. They follow the tables they reference.
• Warnings: existing parts that are not owned warn when they are declared. In this repository's tests, only test_upward_u3_nonprimary_master_fk does: its part references the master below ---. The Cascade may fail when Part table references another Part with renamed foreign keys #1429 renamed part-of-part case is owned and does not warn.

Why it is safe

• Every declaration it refuses would make topo_sort fail, which breaks delete, drop, trace and list_tables(). The changes for existing schemas listed above are why this PR is labeled breaking.
• An existing cycle gives a clear error, and only where an order is needed.
• A scoped sort returns the same tables in an order that respects every dependency between them.
• The current cascade engine is unchanged. It still matches parts to masters by name for part_integrity.
• Cost: declaring a part that references a table other than its master reloads the dependency graph once, for the schemas upstream of the referenced tables. A part that references only its master skips this.

Tests

• tests/unit/test_group_cycle.py (no database):

  • ownership, including renamed references, parallel references, and parts reached through a part that is not owned;
  • the group closure, cycle detection, and the scoped topo_sort;
  • both counterexamples to grouping by name;
  • error messages.

  Two randomized checks run on random dependency graphs:

  • topo_sort succeeds exactly when the graph with each master and its owned parts contracted to one node is acyclic, which is checked with an independent computation of ownership. When it succeeds, every table follows the tables it references, and each master is followed directly by its owned parts.
  • Over sequences of random declarations, group_cycle refuses an owned part exactly when adding it would make topo_sort fail, and no other declaration ever does.

• tests/integration/test_group_cycle.py (MySQL and PostgreSQL):

  • Master → X → Master.Part is refused with the full message. No table is created, and the schema still lists, deletes and sorts.
  • Two groups that reference each other's masters, through the master and through its part, are refused.
  • A cycle across two schemas is refused.
  • An owned part referencing a sibling that is not owned and depends on the master is refused, and the message explains why.
  • A part with a nullable reference to its master and a reference to a table downstream of the master warns, is declared, and is sorted after that table.
  • Allowed references declare without warnings, and trace, cascade and delete work on them.
  • A cycle declared with the check switched off: delete, cascade, trace and list_tables() raise a DataJointError that names the cycle and the part, and no rows are deleted. Unrelated tables keep working. Dropping the part and declaring it again without the reference restores the schema.
  • The warning for a part with no reference to its master, a nullable one, a secondary one, or one only through a part that is not owned. No warning for a renamed reference to the master or for the Cascade may fail when Part table references another Part with renamed foreign keys #1429 renamed part-of-part case.

• Related suites (cascade, trace, diagram, declare, dependencies, schema, alter, foreign keys, virtual modules, autopopulate, unit): 670 passed, 2 skipped.

• Full suite: 1218 passed, 14 skipped. On master the suite gives 1158 passed and 14 skipped; the difference is the 60 new tests.

Docs: the master-part specification (datajoint/datajoint-docs#299) can state the ownership rule and the cycle rule.