fix: return an error for an invalid CSV null_regex instead of panicking - #25261
Merged
Jefffrey merged 1 commit intoSep 15, 2026
Merged
Conversation
`CsvFormat::infer_schema_from_stream` compiled the pattern with
`Regex::new(..).expect("Unable to parse CSV null regex.")`, so a pattern the
regex crate rejects aborted the query task rather than surfacing as a
`DataFusionError`. It is reachable from SQL through any `CREATE EXTERNAL TABLE`
that leaves its columns to schema inference:
CREATE EXTERNAL TABLE t STORED AS CSV LOCATION 'data.csv'
OPTIONS ('format.has_header' 'true', 'format.null_regex' '(');
Compile the regex once before the chunk loop and propagate a failure, which
also stops the pattern being recompiled for every chunk.
Tests: a `statement error` case in `csv_files.slt`. It reproduces the panic
without this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25261 +/- ##
========================================
Coverage 81.88% 81.88%
========================================
Files 1133 1133
Lines 424522 424668 +146
Branches 424522 424668 +146
========================================
+ Hits 347623 347754 +131
- Misses 56285 56296 +11
- Partials 20614 20618 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Jefffrey
approved these changes
Sep 15, 2026
Contributor
|
thanks @Developer1010x |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
A malformed
null_regexpanics the query task instead of returning an error.CsvFormat::infer_schema_from_streamcompiled the pattern withso any pattern the
regexcrate rejects aborted the task. It is reachablestraight from SQL, through a
CREATE EXTERNAL TABLEthat leaves its columns toschema inference:
An invalid regex is a bad option value, not an internal invariant, so it should
come back as an error naming the pattern.
What changes are included in this PR?
The regex is compiled once, before the per-chunk loop, and a failure is
propagated with
exec_datafusion_err!rather than panicking. Hoisting it alsostops the pattern being recompiled for every chunk of the inference stream.
The error text matches the one used on the read side in #25254, so the same bad
option reads the same whichever path hits it first.
What is the testing strategy for this PR?
A
statement error Unable to parse CSV null regexcase indatafusion/sqllogictest/test_files/csv_files.slt.I checked it is not vacuous: with this change reverted, that case fails with the
panic quoted above rather than an error, so the test reproduces the bug.
cargo fmt --check,cargo clippy --all-targetsand the crate's unit tests areclean.
Are there any user-facing changes?
An invalid
null_regexnow produces an error and leaves the session usable,where it previously panicked the task. No API changes.
Independent of #25254 — that one is the read path in
source.rs, this is theinference path in
file_format.rs— but they touch the same crate, so whicheverlands second may want a trivial rebase.