Repository navigation
Public access to hidden attributes (_job_*, _prov): no reader exists #1562
Description
Platform-managed columns — _job_start_time, _job_duration, _job_version, _prov, _singleton — are stored per row and have no public reader. to_arrays('_job_start_time') and proj('_prov') both raise DataJointError: Attribute '...' not found., because Heading.attributes (heading.py:258-267) excludes hidden names and every accessor derives from it.
reference/specs/job-metadata.md documented to_arrays('_job_start_time', ...) as the access path. That call has never worked; the docs are corrected in datajoint/datajoint-docs#288 to describe the SQL workaround until this lands. Supersedes datajoint-docs#1553.
Deferred out of 2.3.4 deliberately — the mechanism touches query machinery and should not be designed under release pressure.
What works today, so the gap is scoped correctly
Measured on MySQL 8.0, not assumed:
Restrict, string form — & "_prov IS NULL" |
works |
Restrict, mapping form — & {"_prov.system": ...} |
silently returns every row — #1561 |
make_sql(["subject_id", "_prov"]) + decode_attribute |
works, composes with restrictions, decodes JSON to a dict on both backends |
to_arrays('_prov'), proj('_prov'), heading['_prov'] |
raise |
So the missing piece is reading values through a supported surface. Restriction already works in its string form, and a correctly decoded read is reachable today in about six lines — which means the design can be judged on ergonomics and safety rather than on feasibility.
Proposed direction: QueryExpression.hidden
A property returning a copy whose heading also carries the hidden attributes:
Analysis.hidden.to_dicts() # includes _job_* and a decoded _prov
Analysis.hidden.proj('_job_version')
Target.insert(Source.hidden) # INSERT ... SELECT carries themReturning a copy is what makes it safe: describe(), alter(), diagram and every un-opted query keep the heading they have now.
The invariant
.hidden changes what is selected and returned. It never changes what is matched on. Join and restriction keys must keep coming from the visible set unconditionally. Three findings make that non-negotiable:
expression.py:398,430build theUSINGlist fromheading.names. Hidden namesakes there reproduce exactly the NATURAL JOIN defect the USING clause was introduced to prevent (specs/job-metadata.md, "Excluding Hidden Attributes from Binary Operators").- Hidden attributes never get a lineage row (
lineage.py:366-369skips any column starting with_), soassert_join_compatibility(condition.py:264-289) would raise "lineage missing on one side" for every join of two metadata-bearing tables. condition.py:441selects restriction keys the same way, and that one fails silently.
Where the work is
heading.py— an opt-in flag pluswith_hidden(); an unconditionally-filtered name list for matching;select()(:641) must carry hidden through,join()(:698) must not._attributesalready holds everything with correctjson/uuid/codec/dtypeflags (:499-503), so nothing new is loaded.expression.py— thehiddenproperty (copy idiom at:1507-1516);proj's validator at:572-576.table.py:877-886— theINSERT ... SELECTbranch currently appends_provby name. Generalizing it is tempting but needs thought: copying_singleton, or_job_*from one table to another, is not meaningful. The_prov-specific carry may be correct because it is specific.
Two defects found on this path
expression.py:578-583is a dead validator.next(a for a in mentions if not self.heading.names)tests whether the heading is empty, not whetherais in it. It never fires, for any name.proj(x='_prov')silently drops the column —_provmatchesrename_pattern, bypassing the positional check, andHeading.select's loop never reaches it. No error, no column. Same family as Mapping restriction naming a hidden attribute silently drops the predicate #1561.
The alternative worth taking seriously
Keep hidden attributes fully hidden and add a narrow documented reader instead — a function taking an expression and attribute names, doing the SELECT and decode_attribute itself. It serves the known consumer (provenance-export, which needs _job_version and _prov per row for audit packets) with decoded values and restriction composition, and touches no query machinery.
The case for it is not weak: attributes' filter currently does five distinct jobs — join-key selection, restriction-key selection, output shaping, insert validation, and DDL round-tripping through describe() ↔ prepare_declare ↔ alter. Splitting them is where silent breakage comes from. The case against is that it is a second way to query, outside the algebra.
Decide between them before implementing.
What must not change
| Behavior | Pinned at |
|---|---|
_job_* absent from heading.names and to_dicts() |
test_hidden_job_metadata.py:184-201 |
| Hidden attributes absent from a join result | test_hidden_job_metadata.py:208-224 |
_prov out of heading, fetch, joins |
test_entry_provenance.py:140-152 |
An author cannot write _prov |
test_entry_provenance.py:155-158 |
Config.heading.primary_key == [] for a singleton |
test_declare.py:421 |
describe() omits _singleton |
test_declare.py:477-490 |
| Users cannot declare an underscore attribute | tests/unit/test_declare_hidden_attribute.py |
.hidden would expose _singleton, so primary_key becomes ['_singleton'] in the opted-in view. Correct, but keys() and restriction run off the primary key — verify on a singleton table.
Acceptance
- Reading returns a decoded
_provdict on both MySQL (json) and PostgreSQL (jsonb); backend-consistent decoding is the likeliest thing to break. - Default
to_dicts()byte-identical to today. A * Bwith_job_*on both sides joins on the visible key only, no lineage error.describe()andalter()round-trip unchanged, including a singleton.test_entry_provenance.py's raw-SQL_raw_provhelper is replaced by the new API — that is the real acceptance test.- Also closes the
declare.py:965error message, which tells users to "useproj()" — currently false.
Activity
dimitri-yatsenko commentedon Oct 1, 2026
#1561 is in scope for this design, not a separate fix.
Same root cause:
Heading.attributes(heading.py:258-267) masks hidden names, so a hidden attribute can be neither matched on (#1561 — a mapping restriction silently drops the predicate and returns every row) nor read back (this issue). Settling one without the other would ship an inconsistent surface: a user who can read_provbut cannot filter by it, or the reverse.The constraint from #1561 that must survive whatever is designed here: a mapping restriction ignores attributes it cannot match by design — that is what lets
Session & keywork whenkeycarries attributes from a more detailed downstream table, andmake()depends on it. The fix is to distinguish absent from_attributesentirely (keep ignoring) from present but hidden (resolve), never to start raising on unmatched keys.dimitri-yatsenko commentedon Oct 1, 2026
#1567 proposes making every platform-managed column declarable through one mechanism in DataJoint notation, by separating the grammar (which should be able to spell
_prov) from the policy (a user may not declare one). It works the same ground as this issue from the other side: #1567 makes hidden attributes uniformly declarable, this one makes them readable, and both turn on where the visible/hidden line is drawn.Worth designing together — a decision here about how an explicitly-named hidden attribute resolves will likely settle how #1567's registry is read back, and vice versa.
dimitri-yatsenko commentedon Oct 18, 2026
#1553 and #1561 cover reading values and the mapping restriction. The other ways users query these columns had not been measured: ordering, aggregation, grouping, computed projections, and joins. The questions behind them are the ones job metadata exists to answer: which rows a given code version computed, the latest computed row in a table, how many rows each code version produced, and how long a step took.
Measured on datajoint
2.3.5.dev8+ga00b89f55(masterat a00b89f) with Python 3.12, against MySQL 8.0.21 and PostgreSQL 16.14. Every outcome below is the same on both backends.Setup:
Src(Manual,provenance.captureon),Comp(Computed,-> Src) andChild(Computed,-> Comp), withjobs.add_job_metadata = Trueandjobs.version_method = "git". Three rows were populated at one commit and three at another. "Works" means the result matched the same values read with raw SQL.Works today
Comp.to_dicts(order_by="_job_start_time DESC", limit=1), and the sameorder_byonkeys()andto_arrays("val")(Src * Comp).to_dicts(order_by="_job_start_time DESC", limit=1)Comp & "_job_start_time > '…'",Comp & "_job_version = '…'",Comp - "_job_version = '…'"(Src * Comp) & "_job_version = '…'"Src & (Comp & "_job_version = '…'"),Child & (Comp & …),Child - (Comp & …)dj.U().aggr(Comp, latest="max(_job_start_time)"), andtotal="sum(_job_duration)"Src.aggr(Comp, n="count(*)", latest="max(_job_start_time)")Comp.proj(ms="_job_duration * 1000"), and& "ms > 0"on the resultComp.proj(v="coalesce(_job_version, '')")Fails today
(Comp * Child).to_dicts(order_by="_job_start_time DESC")_job_start_timeis ambiguous(Comp * Child) & "_job_version = '…'"(Comp & "_job_version = '…'") * ChildComp.aggr(Child, n="count(*)", latest="max(_job_start_time)")Comp.aggr(Child & "_job_version = '…'", n="count(*)")(Comp.proj(x="val * 2") & "x > 0").to_dicts(order_by="_job_start_time DESC")_job_start_timedj.U("_job_version").aggr(Comp, n="count(*)")DataJointError: Attributes ['_job_version'] not found in the group expression.dj.U("_job_version") & CompKeyError: '_job_version'(heading.py:765)Comp.proj(v="(_job_version)")v: the duplicate form of theproj(x='_prov')defect aboveSrc & (Comp & {"_job_version": "…"})What the split shows
The working forms work because the name reaches SQL unchecked.
order_byis quoted and passed through (expression.py:139), and computed expressions inprojandaggrare not checked against the heading. So a query succeeds when the hidden column is in scope in the outermostFROMand unique there. It fails when either condition stops holding:SELECTlists visible names only, so an outerORDER BY _job_start_timehas nothing to refer to.declare.py:362-364), soComp * Childhas two columns for each. A restriction that is valid onCompalone breaks onceCompis joined: in(Comp & "_job_version = '…'") * Child, the predicate is merged unqualified into the join'sWHERE.expression.py:400-402decides whether an operand needs a subquery from visible names only.dj.Uchecks its key againstheading.names(expression.py:1562). The only working way to count rows per code version goes through a computed alias:dj.U("v").aggr(Comp.proj(v="coalesce(_job_version, '')"), n="count(*)")works on both backends.That alias is also a reader: naming a hidden attribute inside a computed expression returns its value today. Naming it is explicit, so it does not break the invariant. It is still a public path the design should either adopt or close.
None of the failing queries needs a hidden attribute as a join key, and the
USINGlist stays visible-only. What they need is for an explicitly named hidden attribute to resolve to a single column that is in scope wherever the algebra puts it.Acceptance: additions
The chosen design,
.hiddenor the reader function, supports these, or this issue states them out of scope:order_by="_job_start_time DESC"withlimitworks onto_dicts,keysandto_arrays. That includes an opted-in expression that the algebra has wrapped in a subquery.dj.U("_job_version").aggr(…)anddj.U("_job_version") & …work on an opted-in expression. Without the opt-in, they raise aDataJointErrorsaying the attribute is hidden, not aKeyError.order_byor an aggregate resolves to one operand, or raises aDataJointErrorthat names the collision. It never produces a backend ambiguity error.(Comp & "_job_version = '…'") * Childreturns the restricted rows. Users can get the latestChildcomputation perComprow, asComp.aggr(Child, latest="max(_job_start_time)")or its opted-in spelling.projrename and duplicate.v="_job_version"andv="(_job_version)"either return the value or raise. They never drop it.Out of scope because it is not specific to hidden attributes:
Comp.proj(latest="max(_job_start_time)")fails exactly asComp.proj(m="max(val)")does, an aggregate withoutGROUP BY;dj.U().aggris the form for that. Time comparisons on_job_start_timeinherit the clock question in #1584.Noticed in passing, not hidden-specific
Src.aggr(Comp & "val <= 30", n="count(*)")returns the 3 matching parents, not all 6 with zero counts, even thoughaggrkeeps non-matching rows by default (exclude_nonmatching=False). The group's restriction lands in theWHEREof theLEFT JOIN(expression.py:1336,1347), which discards the unmatched parents. The result is the same on both backends, and the same with a hidden attribute in the restriction. Not filed.