Repository navigation
migrate.py stamps :uint32:/:int24: markers for core types that do not exist; guide documents a different mapping #1529
Description
Summary
Phase 2 migration stamps core-type markers into column comments that the 2.x type
system does not recognize. The markers are inert, so the effect is cosmetic rather than
destructive — but a migrated schema ends up carrying type labels that mean nothing, and
the migration guide documents a different mapping than the code performs.
The unrecognized types
migrate.py:64-78 maps unsigned integers to uint* core types:
NATIVE_TO_CORE_TYPE = {
# Unsigned integers
"tinyint unsigned": "uint8",
"smallint unsigned": "uint16",
"mediumint unsigned": "uint24",
"int unsigned": "uint32",
"bigint unsigned": "uint64",
...None of those five exist in CORE_TYPES (declare.py:22-47), and all five are rejected
by the declaration parser:
uint8 -> DataJointError: Unsupported attribute type uint8
uint16 -> DataJointError
uint24 -> DataJointError
uint32 -> DataJointError
uint64 -> DataJointError
int24 is also mapped (from mediumint) and does not exist as a core type either,
though it currently slips through the parser for an unrelated reason — see the pattern
anchoring issue filed alongside this one.
This is consistent with the documented intent: explanation/type-system.md:93 states
"Unsigned integer types are not provided. Choose a signed type with sufficient range for
your data." So migrate.py is emitting labels for a category of type that 2.x
deliberately does not have.
Why it is inert rather than fatal
On load, heading.py:506-553 parses the :uint32: marker, sets original_type to
"uint32", then tries to classify it:
try:
category = next(c for c in SPECIAL_TYPES if TYPE_PATTERN[c].match(original_type))
except StopIteration:
if original_type.startswith("external"):
raise DataJointError(...)
# Not a special type - that's fine, could be native passthrough
category = NoneSo it lands in the category = None branch and is silently dropped. The column keeps
its database type (int unsigned), which matches INTEGER, so values still round-trip
correctly as integers. Nothing raises and no data is affected.
What is left is a schema whose column comments claim a type the user cannot write in a
definition, and which original_type will surface in displays.
Contradiction with the guide
how-to/migrate-to-v20.md:1080-1084 documents the opposite mapping:
| guide says | code does |
|---|---|
int unsigned → int64 |
int unsigned → uint32 |
smallint unsigned → int32 |
smallint unsigned → uint16 |
tinyint unsigned → int16 |
tinyint unsigned → uint8 |
bigint unsigned → int64 |
bigint unsigned → uint64 |
The guide's version is the one that matches the type system — widening to a signed type
that can hold the full unsigned range is lossless and produces a type that actually
exists. migrate.py:2285 (- int unsigned→COMMENT ':uint32:...'``) documents the
code's behaviour, so the two halves of the guide disagree with each other as well.
Proposed fix
Decide which mapping is correct and make all three agree — code, guide, and the
migrate.py docstring. My reading is that the guide is right:
"tinyint unsigned": "int16",
"smallint unsigned": "int32",
"mediumint unsigned": "int32",
"int unsigned": "int64",
"bigint unsigned": "int64", # note: values above 2^63-1 do not fit
"mediumint": "int32", # int24 is not a core type eitherbigint unsigned is the one case with no lossless signed target, so it needs an explicit
decision — either accept the ceiling or leave those columns unlabeled with a warning.
Worth adding a test that every value in NATIVE_TO_CORE_TYPE is accepted by
match_type; that assertion would have caught this.
Activity
fix: validate attribute types at declaration and insert (#1527, #1528, …