Skip to content

fix(bigquery): keep the Arrow schema for empty results - #18655

Open
ohmayr wants to merge 1 commit into
mainfrom
fix/bigquery-arrow-empty-schema
Open

ohmayr wants to merge 1 commit into
mainfrom
fix/bigquery-arrow-empty-schema

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Yields one empty RecordBatch with the query's schema when Arrow results have no rows, so to_arrow() no longer returns a table without columns.

  • Uses the Arrow schema BigQuery sent, else the schema from jobs.getQueryResults, else no columns, like _EmptyRowIterator.
  • Covers results that fit in the first page, an empty stream, and an empty destination table.
  • Adds unit tests and system tests for empty queries and DDL/DML statements.

@ohmayr
ohmayr requested review from a team as code owners October 13, 2026 09:17
@ohmayr
ohmayr requested review from sindhuvy and removed request for a team October 13, 2026 09:17

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that Arrow queries returning zero rows still yield a single empty batch so that callers can retrieve the schema. It introduces a helper function _empty_batch and updates download_arrow_from_job to track whether any batches have been yielded, along with adding corresponding unit and system tests. A review comment suggests guarding the empty batch yield when reader is None with if not yielded to prevent appending an extra empty batch to a non-empty result set.

Comment on lines 337 to 340
if reader is None:
# An empty destination table has no streams.
yield _empty_batch(pa_schema, row_iterator._schema)
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If reader is None but yielded is already True (e.g., some batches were already yielded in Step 1), yielding an empty batch here would append an extra empty batch to a non-empty result set. We should guard this with if not yielded: to ensure we only yield an empty batch when no rows have been yielded at all.

Suggested change
if reader is None:
# An empty destination table has no streams.
yield _empty_batch(pa_schema, row_iterator._schema)
return
if reader is None:
# An empty destination table has no streams.
if not yielded:
yield _empty_batch(pa_schema, row_iterator._schema)
return

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reader is only None on the explicit-destination path, and explicit destinations always go through jobs.insert, where we wait with getQueryResults(maxResults=0) so there's no inline Arrow batch. yielded is always False there.

@ohmayr
ohmayr added this pull request to stack #18648 October 13, 2026 09:49
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch from 825fed7 to b501684 Compare October 13, 2026 10:02
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch 3 times, most recently from 05834ef to d77849b Compare October 16, 2026 01:16
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch from d77849b to dbc30a0 Compare October 16, 2026 21:45
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch 2 times, most recently from 0d21c36 to 4b1ddd1 Compare October 19, 2026 23:26
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch 3 times, most recently from a7b0ad3 to 263c70f Compare October 20, 2026 00:49
@ohmayr
ohmayr requested review from tswast and removed request for sindhuvy October 20, 2026 20:35
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch from 263c70f to d7a6b4c Compare October 20, 2026 22:38
@tswast
tswast force-pushed the fix/bigquery-arrow-empty-schema branch from d7a6b4c to 32b3c10 Compare October 21, 2026 16:01
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch from 32b3c10 to f277a64 Compare October 21, 2026 18:55
# this package's table module.
from google.cloud.bigquery import _pandas_helpers

pa_schema = _pandas_helpers.bq_to_arrow_schema(bq_schema)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? Do we not get an arrow schema in the jobs.query response if the query completes with empty results.

@ohmayr ohmayr Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed b/573768143. Until this is addressed, we'll need to ensure we return an empty table when the response is empty.

I believe the empty schema case (this function) is not relevant to jobs.query but to the case where ReadRows is called after jobs.insert.

Correction: you're right that jobs.query returns arrowSchema when the query completes, even with 0 rows, so this isn't used there. It's for empty results read from the job's _default stream: ReadRows sends no messages at all for 0 rows, so there's no schema. That happens after jobs.insert with an anonymous destination, and also after jobs.query when the query doesn't finish in time and we wait. Checked all of these against BigQuery. Explicit destinations get the schema from the read session.

Base automatically changed from feat/bigquery-arrow-explicit-destination to main October 21, 2026 22:42
A query that returned no rows with ARROW yielded no batches, so
to_arrow() fell back to the iterator's schema, which jobs.query omits
for ARROW, and returned a table without columns. jobs.query sends
arrowSchema even for zero rows, but the default job stream sends
nothing for them.

to_arrow_iterable() now yields one empty RecordBatch if no rows arrive,
like _EmptyRowIterator already does. It has the Arrow schema from the
first page or the read session; otherwise the schema from
jobs.getQueryResults, which the jobs.insert path has; otherwise no
columns. to_arrow() and pyarrow.Table.from_batches() keep that schema.
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-empty-schema branch from f277a64 to 450fe2f Compare October 21, 2026 22:42
@ohmayr ohmayr added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 21, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants