Validating Date/Datetime columns imported as strings

[adrianstone]

Hi all,

I'm trying to better understand the intended workflow for validating Date/Datetime columns with Dataframely.

In my case, I'm importing data from Excel. If a date column contains a single invalid value (e.g. "banana"), pl.read_excel() imports the entire column as String, while valid dates are represented as datetime strings.

For example:

import polars as pl
import dataframely as dy


class TestSchema(dy.Schema):
    value = dy.Datetime(nullable=True)


df = pl.DataFrame(
    {
        "value": [
            "banana",
            "2026-03-18 00:00:00",
            "2026-04-17 00:00:00",
            None,
        ]
    }
)

good, failures = TestSchema.filter(
    df,
    cast=True,
)

print(failures.invalid())

Output:

shape: (3, 1)
┌─────────────────────┐
│ value               │
│ ---                 │
│ str                 │
╞═════════════════════╡
│ banana              │
│ 2026-03-18 00:00:00 │
│ 2026-04-17 00:00:00 │
└─────────────────────┘

In this case, all non-null values are reported as invalid.

I was wondering if I'm missing a step somewhere. I could pre-convert the column before validation, but that would replace invalid values with null:

pl.col("value").str.to_datetime(strict=False)

My goal is to identify invalid values while still validating that date/datetime strings are parseable.

Given that, I'm not sure if I'm misunderstanding how Dataframely is intended to handle mixed-value columns that are imported as String.

Is preprocessing the expected approach here, or is there a recommended Dataframely approach for handling this type of input?

Any insight would be appreciated.

Thank you

[David Jang (ChangkeunJ)]

Preprocessing is the expected approach, but the reason all three rows come back invalid is worth spelling out, because it isn't the reason it looks like.

cast=True runs the frame through match_to_schema(..., casting="lenient"), and lenient casting is literally this, from _match_to_schema.py:

def _cast_if_required(expr, current_dtype, column, *, strict=True):
    if column.validate_dtype(current_dtype):
        return expr
    return expr.cast(column.dtype, strict=strict)

So what runs is pl.col("value").cast(pl.Datetime, strict=False). That's cast, not str.to_datetime, and in polars those aren't the same thing at all. cast is not a date parser. Anything it can't turn into a Datetime becomes null under strict=False, and then dataframely adds this rule per column when with_cast is on, in _build_rules:

DtypeCastRule(
    pl.col(col_name).is_null()
    == pl.col(f"{ORIGINAL_COLUMN_PREFIX}{col_name}").is_null()
)

Non-null before the cast and null after counts as a cast failure. Your two well-formed datetime strings didn't survive cast any more than "banana" did, so all three trip that rule. failures.counts() should show them all under value|dtype, which is a quick way to confirm it. And it's why the failures frame is still str, since it reports the originals.

For what you actually want, which is seeing the bad values rather than silently nulling them, I'd leave the column as a String in the schema and put the parse in a check:

class TestSchema(dy.Schema):
    value = dy.String(
        nullable=True,
        check=lambda expr: expr.str.to_datetime(strict=False).is_not_null(),
    )

"banana" fails the check, the genuine null stays valid because nullable=True, and failures.invalid() hands you back the offending text. Then do the real cast once you're through validation:

good, failures = TestSchema.filter(df)
typed = good.with_columns(pl.col("value").str.to_datetime())

If you'd rather the schema hold the actual Datetime type, the alternative is to parse first with str.to_datetime(strict=False) and declare dy.Datetime(nullable=False), so unparseable values become null and get caught by the nullability rule instead. That works, but you lose the ability to tell a genuinely missing value from one that failed to parse, and invalid() shows you a null where you wanted to see "banana".

[adrianstone]

Thank you for the explanation. The distinction around pl.col("value").cast(pl.Datetime, strict=False) clarified it for me.

My use case is validating Excel workbooks and generating reviewer-facing findings. After considering the approaches you outlined, I ended up handling date parsing as a preprocessing step.

I considered using a String field with a parsing check, but that would have required rewriting rules that are currently defined against the expected data types rather than strings.

Instead, I compare the data type inferred from the Excel import to the type defined in the Dataframely schema. When they don't align, I attempt the conversion and capture any resulting parsing failures.

I then merge those findings with the validation failures returned by filter() as part of my reporting pipeline. This lets me preserve the original invalid values while keeping the schema definitions unchanged.

For my use case, that's a reasonable tradeoff since the primary goal is to identify and reconcile workbook issues before any downstream processing occurs.