Skip to content

Fix strict evaluation for negative record counts #3831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevinjqliu merged 1 commit into from
Aug 23, 2026
+7 −5
Merged

Fix strict evaluation for negative record counts #3831

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions pyiceberg/expressions/visitors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1527,12 +1527,14 @@ def eval(self, file: DataFile) -> bool:
Returns: false if the file may contain any row that doesn't match
the expression, true otherwise.
"""
if file.record_count <= 0:
# Older version don't correctly implement record count from avro file and thus
# set record count -1 when importing avro tables to iceberg tables. This should
# be updated once we implemented and set correct record count.
if file.record_count == 0:
return ROWS_MUST_MATCH

if file.record_count < 0:
# Older versions set the record count to -1 when importing Avro tables.
# Treat an unknown count conservatively rather than as an empty file.
return ROWS_MIGHT_NOT_MATCH

return visit(self.expr, _StrictMetricsEvaluationVisitor(self.struct, file))


Expand Down
2 changes: 1 addition & 1 deletion tests/expressions/test_evaluator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1523,7 +1523,7 @@ def test_metrics_evaluator_record_count_short_circuits() -> None:
)
assert _InclusiveMetricsEvaluator(schema, EqualTo("x", 10)).eval(negative_record_file) is ROWS_MIGHT_MATCH
assert _StrictMetricsEvaluator(schema, EqualTo("x", 10)).eval(zero_record_file) is ROWS_MUST_MATCH
assert _StrictMetricsEvaluator(schema, EqualTo("x", 10)).eval(negative_record_file) is ROWS_MUST_MATCH
assert _StrictMetricsEvaluator(schema, EqualTo("x", 10)).eval(negative_record_file) is ROWS_MIGHT_NOT_MATCH


def test_strict_not(schema_data_file: Schema, strict_data_file_1: DataFile) -> None:
Expand Down