Skip to content

Global transform: false loads simple values as NULL #81

Closed

Description

@fabbiot

With the global option transform: false (added in #72), simple values are loaded as NULL. The same file loads correctly with the default config.

Version: 0.14.0 and current main (f35cc58), xmlschema 4.3.2, DuckDB in memory. Also seen on PostgreSQL 18.

person.xsd:

<?xml version="1.0"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="people"><xs:complexType><xs:sequence>
    <xs:element name="person" type="personType" maxOccurs="unbounded"/>
  </xs:sequence></xs:complexType></xs:element>
  <xs:complexType name="personType"><xs:sequence>
    <xs:element name="name" type="xs:string"/>
    <xs:element name="age" type="xs:int"/>
    <xs:element name="address" type="addressType"/>
  </xs:sequence></xs:complexType>
  <xs:complexType name="addressType"><xs:sequence>
    <xs:element name="city" type="xs:string"/>
  </xs:sequence></xs:complexType>
</xs:schema>

people.xml:

<?xml version="1.0"?>
<people><person><name>Ada</name><age>36</age><address><city>London</city></address></person></people>

Script (prints each table's non-key columns):

import sys
import sqlalchemy
from xml2db import DataModel

for config in ({}, {"transform": False}):
    engine = sqlalchemy.create_engine("duckdb:///:memory:")
    model = DataModel(sys.argv[1], short_name="people", db_engine=engine, model_config=config)
    model.parse_xml(sys.argv[2]).insert_into_target_tables()
    with engine.connect() as conn:
        tables = [t for t in sqlalchemy.inspect(engine).get_table_names() if not t.startswith("temp")]
        rows = {t: conn.execute(sqlalchemy.text(f'select * from "{t}"')).mappings().all() for t in sorted(tables)}
    print(config, {t: [{k: v for k, v in r.items() if not k.startswith(("pk_", "fk_", "xml2db"))} for r in rs] for t, rs in rows.items()})

Output of python repro.py person.xsd people.xml:

{} {'people': [{}], 'people_person': [{}], 'person': [{'name': 'Ada', 'age': 36, 'address_city': 'London'}]}
{'transform': False} {'address': [{'city': None}], 'people': [{}], 'people_person': [{}], 'person': [{'name': None, 'age': None}]}

Expected with transform: false: person with name='Ada', age=36, and address with city='London'. The import reports success, so the loss is silent.

Setting transform: false per field instead (tables.person.fields.address.transform: false) works as expected.

A possible lead: in DataModelTransformedTable._get_field_transform, the global flag also turns off the default "join" for simple columns, not only elevate, and parsed simple values may then not be converted to scalars before insertion. I have not confirmed this.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions