Skip to content

Commit 6f0775e

Browse files
test(diagram): define groups by ownership, as the revised spec does
Follow the revised traversal specification (datajoint-docs#249): - A1 is ownership: a part is owned if an FK in its primary key references the master or an owned part of the same master, renames allowed. Its owning attributes come from that chain, preferring an unrenamed -> master and otherwise the first owning reference. This replaces the strict reading and the a1 switch; a part that references its master only below the line is no longer in its group. - The lift goes through the owning attributes, and a group iterates if and only if some FK inside it does not map owning attributes onto owning attributes. The relabel rule is the revised one. - The shape catalog gains the rewritten G6 (a part its master does not own) and G7 (an owned part-of-part with renamed attributes, #1429). The three xfails for the spec's iteration and relabel rules are gone: the docs now state the model's rules, and the tests assert them. Two strict xfails remain for T8: topo_sort still groups every part with its master by name, while the spec makes a part that is not owned a plain table (#1619). test_expand_spec.py opens with a description of the test layers and of the Hypothesis profiles.
1 parent 38a8356 commit 6f0775e

3 files changed

Lines changed: 435 additions & 620 deletions

File tree

‎tests/unit/expand_model.py‎

Lines changed: 72 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
The module also models parts of the specification's "Computation" section, so
2323
that its claims can be checked against the definitions: the table-level reach
2424
(the node set of an expanded diagram), the one pass over the group quotient with
25-
its iterating groups, and the conditions under which R1 may copy a condition or
25+
its iterating groups, and the rules under which R1 may copy a condition or
2626
relabel materialized keys.
2727
2828
Model
@@ -32,10 +32,10 @@
3232
Parallel FKs between the same two tables are separate edges.
3333
- A2: an FK with any NULL column references nothing, as in SQL's MATCH SIMPLE.
3434
FKs can share columns, so partial NULLs occur.
35-
- A1, the owning reference: a part's heading contains its master's primary key,
36-
unrenamed and non-null, inherited through ``-> master`` directly or through a
37-
sibling part. A part that violates A1 is traversed as a plain table.
38-
``Schema(..., a1="chain")`` selects a generalized A1, described in ``Schema``.
35+
- A1, owned parts: a part is owned by its master if an FK in its primary key
36+
references the master or an owned part of the same master, renames allowed.
37+
The chain maps the part's owning attributes onto the master's key. A master
38+
and its owned parts form its group; a part that is not owned is a plain table.
3939
- A database is ``{table: {pk_tuple: row_dict}}``, where ``pk_tuple`` lists the
4040
primary-key values in the order of ``TableSpec.pk``.
4141
"""
@@ -130,16 +130,6 @@ class Schema:
130130
Tables in declaration order.
131131
fks : iterable of FK
132132
Foreign keys. Each child's FKs are listed in declaration order.
133-
a1 : {"strict", "chain"}, optional
134-
Which owning reference makes a part a member of its master's group.
135-
136-
- ``"strict"`` (the specification): an FK to the master, or to a sibling
137-
that is a member, that carries the master's key unrenamed and non-null.
138-
- ``"chain"`` (a generalization, for #1429): also a primary-key FK, with
139-
renames allowed, to the master or to a member whose owning attributes
140-
lie in its primary key. The owning attributes are then composed along
141-
the chain. When a part has several such FKs, an unrenamed one wins,
142-
and otherwise the first in declaration order.
143133
144134
Notes
145135
-----
@@ -149,33 +139,34 @@ class Schema:
149139
FK that includes it is nullable, as in ``declare.py``, where an FK adds
150140
only the attributes not declared yet. Primary-key attributes and
151141
attributes outside every FK are never NULL.
152-
- **Groups.** Each master that has parts forms a group with its member
153-
parts. ``group[t]`` names the master of table ``t`` for the master itself
154-
and for its members. A part that is not a member is in no group.
142+
- **Owned parts and groups** (A1). A part is owned if an FK in its primary
143+
key (every attribute of the FK in the primary key, as ``Dependencies``
144+
marks it ``primary``) references the master or an owned part of the same
145+
master, renamed or not. Each master that has parts forms a group with its
146+
owned parts; ``group[t]`` names the master of table ``t`` for the master
147+
and for its owned parts. A part that is not owned is in no group.
155148
- **Owning attributes.** ``owning[t]`` maps each attribute of the master's
156-
primary key to the attribute of member ``t`` that holds it: the identity
157-
for the master, and for every member under ``a1="strict"``.
149+
key to the attribute of ``t`` that the owning chain maps onto it: the
150+
identity for the master. If a part has several owning references, an
151+
unrenamed ``-> master`` wins, and otherwise the first in its definition.
158152
"""
159153

160-
def __init__(self, tables: Iterable[TableSpec], fks: Iterable[FK], a1: str = "strict") -> None:
161-
if a1 not in ("strict", "chain"):
162-
raise ValueError(f"a1 must be 'strict' or 'chain', not {a1!r}")
163-
self.a1 = a1
154+
def __init__(self, tables: Iterable[TableSpec], fks: Iterable[FK]) -> None:
164155
self.tables = tuple(tables)
165156
self.fks = tuple(fks)
166157
self._by_name = {t.name: t for t in self.tables}
167158
if len(self._by_name) != len(self.tables):
168159
raise ValueError("table names must be unique")
169160
self._validate()
170161
self.nullable = {t.name: self._nullable_attributes(t) for t in self.tables}
171-
self.a1_parts = {}
162+
self.owned_parts = {}
172163
self.group = {}
173164
self.owning = {}
174165
for t in self.tables:
175166
parts = [p for p in self.tables if p.master == t.name]
176167
if parts:
177-
owning = self._find_members(t, parts)
178-
self.a1_parts[t.name] = tuple(p.name for p in parts if p.name in owning)
168+
owning = self._find_owned(t, parts)
169+
self.owned_parts[t.name] = tuple(p.name for p in parts if p.name in owning)
179170
self.owning.update(owning)
180171
self.group.update({name: t.name for name in owning})
181172

@@ -187,8 +178,7 @@ def __repr__(self) -> str:
187178
for fk in self.fks:
188179
null = ", nullable=True" if fk.nullable else ""
189180
lines.append(f" FK({fk.child!r}, {fk.parent!r}, {dict(fk.attr_map)!r}{null}),")
190-
a1 = "" if self.a1 == "strict" else f" a1={self.a1!r},\n"
191-
return "\n".join(lines) + "\n" + a1 + ")"
181+
return "\n".join(lines) + "\n)"
192182

193183
def table(self, name: str) -> TableSpec:
194184
"""Return the table named ``name``."""
@@ -247,46 +237,20 @@ def _nullable_attributes(self, table: TableSpec) -> frozenset[str]:
247237
nullable.add(attr)
248238
return frozenset(nullable)
249239

250-
def _find_members(self, master: TableSpec, parts: list[TableSpec]) -> dict[str, dict[str, str]]:
251-
# The strict owning reference is found first, to a fixed point, so that
252-
# the result does not depend on the order in which the parts are listed.
253-
# Under a1="chain", primary-key FKs with renames are then accepted for
254-
# the parts left over.
240+
def _find_owned(self, master: TableSpec, parts: list[TableSpec]) -> dict[str, dict[str, str]]:
241+
# Parts are taken in declaration order, so an owning reference to another
242+
# part is decided after that part (an FK references a table declared earlier).
255243
owning = {master.name: {a: a for a in master.pk}}
256-
rules = [self._strict_owning] + ([self._chain_owning] if self.a1 == "chain" else [])
257-
for rule in rules:
258-
changed = True
259-
while changed:
260-
changed = False
261-
for part in parts:
262-
if part.name in owning:
263-
continue
264-
for fk in self.fks_of(part.name):
265-
found = rule(fk, master.pk, owning)
266-
if found is not None:
267-
owning[part.name] = found
268-
changed = True
269-
break
244+
for part in parts:
245+
chains = [fk for fk in self.fks_of(part.name) if fk.parent in owning and self.is_primary(fk)]
246+
if not chains:
247+
continue
248+
unrenamed = [fk for fk in chains if fk.parent == master.name and not fk.renamed]
249+
fk = (unrenamed or chains)[0]
250+
by_parent_attr = {p: c for c, p in fk.attr_map.items()}
251+
owning[part.name] = {a: by_parent_attr[owning[fk.parent][a]] for a in master.pk}
270252
return owning
271253

272-
def _strict_owning(self, fk: FK, master_key: tuple, owning: dict) -> dict | None:
273-
# Unrenamed and non-null: the part holds the master's key under its names.
274-
if fk.parent == fk.child or fk.parent not in owning or any(owning[fk.parent][a] != a for a in master_key):
275-
return None
276-
if all(fk.attr_map.get(a) == a and a not in self.nullable[fk.child] for a in master_key):
277-
return {a: a for a in master_key}
278-
return None
279-
280-
def _chain_owning(self, fk: FK, master_key: tuple, owning: dict) -> dict | None:
281-
# A primary-key FK to a member whose owning attributes it carries, renamed or not.
282-
if fk.parent == fk.child or fk.parent not in owning or not self.is_primary(fk):
283-
return None
284-
by_parent_attr = {p: c for c, p in fk.attr_map.items()}
285-
parent_owning = owning[fk.parent]
286-
if not all(parent_owning[a] in by_parent_attr for a in master_key):
287-
return None # the parent's owning attributes are not in its primary key
288-
return {a: by_parent_attr[parent_owning[a]] for a in master_key}
289-
290254

291255
# =============================================================================
292256
# Databases and references
@@ -356,9 +320,9 @@ def master_of(schema: Schema, table: str, row: Mapping) -> Row | None:
356320
"""
357321
``master(r)``: identifier of the master row of a row in a group.
358322
359-
The master row is named by the row's owning attributes, which A1 guarantees
360-
are present and non-null. Returns None for a row in no group. The identifier
361-
is returned whether or not the master row exists.
323+
The master row is named by the row's owning attributes, which lie in the
324+
primary key and so are never NULL. Returns None for a row in no group. The
325+
identifier is returned whether or not the master row exists.
362326
"""
363327
attrs = schema.owning_attributes(table)
364328
if attrs is None:
@@ -406,8 +370,8 @@ def group_whole(schema: Schema, db: DB) -> bool:
406370
delete must remove each master-part instance whole or not at all, which
407371
``RowGraph.group_closed`` checks on the deleted rows. Removing one part row
408372
of an instance leaves a database that still passes this check, yet breaks
409-
P2. Under A1 the check also follows from ``ri_ok``, since each part row
410-
references its master directly or through a sibling.
373+
P2. For owned parts the check also follows from ``ri_ok``, since each part
374+
row references its master along its owning chain.
411375
"""
412376
for table, master in schema.group.items():
413377
if table == master:
@@ -423,8 +387,8 @@ def quotient(schema: Schema) -> nx.DiGraph:
423387
"""
424388
The group quotient Q: the FK graph with each master-part group contracted to one node.
425389
426-
FKs between two tables of the same group are left out. A part that violates
427-
A1 is in no group and stays its own node. A self-referencing FK is a
390+
FKs between two tables of the same group are left out. A part that is not
391+
owned is in no group and stays its own node. A self-referencing FK is a
428392
self-loop, which counts as a cycle.
429393
"""
430394
node = {t.name: schema.group.get(t.name, t.name) for t in schema.tables}
@@ -447,7 +411,7 @@ def table_reach(schema: Schema, tables: Iterable[str], direction: str = "down",
447411
The node set of an expanded diagram: the table-level reach of ``tables``.
448412
449413
The tables reached by following FKs in the direction of travel, together
450-
with the master and all member parts of every group that has a reached
414+
with the master and all owned parts of every group that has a reached
451415
member. ``group_step=False`` leaves the group step out (``D1``). For
452416
``"both"``, it is the upward reach of the downward reach.
453417
"""
@@ -465,7 +429,7 @@ def table_reach(schema: Schema, tables: Iterable[str], direction: str = "down",
465429
nxt = {fk.parent for fk in schema.fks if fk.child == t}
466430
if group_step and t in schema.group:
467431
master = schema.group[t]
468-
nxt |= {master, *schema.a1_parts[master]}
432+
nxt |= {master, *schema.owned_parts[master]}
469433
for n in nxt - seen:
470434
seen.add(n)
471435
stack.append(n)
@@ -497,8 +461,8 @@ class RowGraph:
497461
- ``child(r) = {c : ref_e(c) = r}`` and ``parent(r) = {ref_e(r)}``, over
498462
every FK edge e.
499463
- ``inst(r)`` is the master-part instance of a row in a group: the master
500-
row and every member part row that carries the same master key. For a row
501-
in no group, ``inst(r) = {r}``.
464+
row and every owned part row whose owning attributes hold the same master
465+
key. For a row in no group, ``inst(r) = {r}``.
502466
- ``down = child ∪ inst`` and ``up = parent ∪ inst``, so ``up`` is the
503467
converse of ``down``.
504468
"""
@@ -635,41 +599,34 @@ def group_closed(self, rows: Iterable[Row]) -> bool:
635599

636600
def preserves_instance(schema: Schema, fk: FK) -> bool:
637601
"""
638-
True if an FK inside a group always references a row of the same instance.
602+
True if an FK inside a group maps the referencing part's owning attributes
603+
onto the referenced table's owning attributes, so it stays in the instance.
639604
640-
It does when it maps the child's owning attribute of each master-key
641-
attribute onto the parent's owning attribute of the same one.
605+
A group iterates if and only if some FK inside it does not.
642606
"""
643607
master = schema.group[fk.child]
644608
child_owning, parent_owning = schema.owning[fk.child], schema.owning[fk.parent]
645609
return all(fk.attr_map.get(child_owning[a]) == parent_owning[a] for a in schema.table(master).pk)
646610

647611

648-
def renames_master_key(schema: Schema, fk: FK) -> bool:
649-
"""The specification's test for an iterating group: the FK renames an attribute of the master's key."""
650-
parent_owning = set(schema.owning[fk.parent].values())
651-
return any(c != p for c, p in fk.attr_map.items() if p in parent_owning)
652-
653-
654-
ITERATION_TESTS = {
655-
"spec": renames_master_key,
656-
"exact": lambda schema, fk: not preserves_instance(schema, fk),
657-
}
612+
def iterates(schema: Schema, master: str) -> bool:
613+
"""Whether the group of ``master`` iterates: some FK inside it does not preserve the instance."""
614+
return not all(preserves_instance(schema, fk) for fk in schema.intra_group_fks(master))
658615

659616

660617
def one_pass(
661-
schema: Schema, db: DB, seed: Iterable[Row], direction: str = "down", iterate: str | None = "exact"
618+
schema: Schema, db: DB, seed: Iterable[Row], direction: str = "down", iterate: bool = True
662619
) -> tuple[frozenset[Row], dict]:
663620
"""
664621
The engine's computation, evaluated on rows: one visit per node of Q.
665622
666623
The nodes of Q are visited in topological order for ``"down"`` and in
667624
reverse order for ``"up"``. A plain table takes its seed rows and the rows
668625
that R1 reaches from its neighbors on the incoming side. A group takes the
669-
master keys of its seed rows and of the rows R1 reaches in any of its
670-
tables, then all rows of those instances. A group that iterates repeats the
671-
group step, following the FKs inside the group, until no new master key
672-
appears.
626+
master keys of its seed rows and of the rows R1 reaches in any of its tables
627+
(the lift, through the owning attributes), then all rows of those instances
628+
(the re-expansion). A group that iterates repeats the group step, following
629+
the FKs inside the group, until no new master key appears.
673630
674631
Parameters
675632
----------
@@ -679,11 +636,9 @@ def one_pass(
679636
The seed rows.
680637
direction : {"down", "up"}
681638
The direction of travel.
682-
iterate : {"exact", "spec"} or None
683-
Which groups iterate: those with an FK inside the group that does not
684-
preserve the instance (``"exact"``), those with an FK inside the group
685-
that renames the master's key (``"spec"``, the rule in ``diagram.md``),
686-
or none.
639+
iterate : bool
640+
Whether the groups that ``iterates`` selects repeat the group step. With
641+
False, every group takes one step.
687642
688643
Returns
689644
-------
@@ -704,7 +659,6 @@ def one_pass(
704659
members = defaultdict(list)
705660
for t in schema.tables:
706661
members[node_of[t.name]].append(t.name)
707-
test = ITERATION_TESTS[iterate] if iterate else None
708662

709663
def rows_of(table):
710664
return {(table, k) for k in db[table]}
@@ -727,16 +681,16 @@ def crossing(fk, reached):
727681
outside = fk.parent if direction == "down" else fk.child
728682
if inside in mem and node_of[outside] != node:
729683
incoming |= crossing(fk, reached)
730-
if node not in schema.a1_parts:
684+
if node not in schema.owned_parts:
731685
reached |= incoming
732686
continue
733687
inner = schema.intra_group_fks(node)
734-
iterates = test is not None and any(test(schema, fk) for fk in inner)
688+
repeat = iterate and iterates(schema, node)
735689
keys = {graph.master(r) for r in incoming}
736690
history = [set(keys)]
737691
while True:
738692
instance = {r for t in mem for r in rows_of(t) if graph.master(r) in keys}
739-
if not iterates:
693+
if not repeat:
740694
break
741695
step = set().union(*(crossing(fk, instance) for fk in inner))
742696
grown = keys | {graph.master(r) for r in step}
@@ -866,9 +820,20 @@ def relabel_up(fk: FK, keys: list[dict]) -> list[dict]:
866820

867821

868822
def relabel_down_allowed(schema: Schema, fk: FK, keys: list[dict]) -> bool:
869-
"""The Seeds rule for relabeling downward: the FK has no nullable column, or every key names all its columns."""
870-
parent_attrs = set(fk.attr_map.values())
871-
return not set(fk.attr_map) & schema.nullable[fk.child] or all(parent_attrs <= set(key) for key in keys)
823+
"""
824+
The Seeds rule for relabeling parent keys downward.
825+
826+
Every key names, among the parent's attributes, only attributes the FK
827+
carries; and if the FK has a nullable column, every key names all of the
828+
FK's attributes, none of them NULL.
829+
"""
830+
parent = schema.table(fk.parent)
831+
carried = set(fk.attr_map.values())
832+
if any(set(key) & set(parent.heading) - carried for key in keys):
833+
return False
834+
if not set(fk.attr_map) & schema.nullable[fk.child]:
835+
return True
836+
return all(carried <= set(key) and all(key[a] is not None for a in carried) for key in keys)
872837

873838

874839
# =============================================================================

0 commit comments

Comments
 (0)