Skip to content

[C++][Python] Use timestamp_parsers as a fallback for CSV time columns #54651

Open

Description

@pearu

Describe the enhancement requested

CSV columns typed as time32 or time64 are parsed only from the ISO-8601 format, and ConvertOptions.timestamp_parsers applies only to timestamp columns:

import io
import pyarrow as pa
from pyarrow import csv

convert_options = csv.ConvertOptions(
    column_types={"t": pa.time32("s")}, timestamp_parsers=["%H.%M.%S"]
)
print(csv.read_csv(io.BytesIO(b"t\n14.30.00\n"), convert_options=convert_options))
pyarrow.lib.ArrowInvalid: In CSV column #0: CSV conversion error to time32[s]: invalid value '14.30.00'

The workaround is to read the column as a timestamp and cast it to a time. I propose that time columns try timestamp_parsers after the ISO-8601 parser fails, and take the time of day of the parsed timestamp, as a cast from timestamp to time does.

An implementation exists in #50146, which is being refactored into smaller PRs.

Part of #41488.

Component(s)

C++, Python

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions