Skip to content

Render ARRAY and ITERABLE fields that are not backed by a List - #39750

Open
PDGGK wants to merge 2 commits into
apache:masterfrom
PDGGK:fix-rowtostring-iterable
Open

Render ARRAY and ITERABLE fields that are not backed by a List#39750
PDGGK wants to merge 2 commits into
apache:masterfrom
PDGGK:fix-rowtostring-iterable

Conversation

@PDGGK

@PDGGK PDGGK commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #39749.

Row#toString throws for an ITERABLE field holding a plain Iterable:

Schema s = Schema.builder().addStringField("k").addIterableField("vals", FieldType.STRING).build();
Row.withSchema(s).attachValues("k1", () -> list.iterator()).toString();
// IllegalArgumentException: value type is '...' for field type 'ITERABLE'

toPrettyFieldValueString demanded a List before iterating. An ITERABLE field declares an Iterable, so the guard was stricter than the type it guards, and the branch below only ever iterates and counts — both fine from an Iterable once materialised.

This is reached from Row#toString, so refusing to render one field takes out logging and debugger output for every field beside it. That is a poor trade for a stricter check in a renderer.

The guard now requires Iterable; a value that is neither still throws the same exception.

On where this belongs

A reviewer could reasonably hold that a materialised Row should always store a List for ARRAY/ITERABLE, and that a non-List reaching here means a producer is at fault — ByteBuddyUtils.transformContainer hands back a Collections2.TransformedCollection for a Set-typed POJO field, for instance. I think that is worth looking at separately and have not touched it.

What makes this worth fixing on its own is that the reproducer needs only Schema.builder, attachValues and toString() — no POJO, no registry, no pipeline — so the renderer is reachable with a merely-Iterable value through plain public API. And a toString() that throws is difficult to justify whatever produced the value.

I should be straight about the blast radius rather than inflate it: I could not construct a case where a non-List reaches user code inside a running pipeline. Coder round-trips materialise, and Convert.toRows() / @Element Row both yield Lists. The demonstrated exposure is direct API use.

Tests

case on master
bare Iterable on an ITERABLE field fails with the IllegalArgumentException above
ordinary List on an ARRAY field passes — the control

The first also asserts !(row.getValue("vals") instanceof List) before rendering, so if the value ever starts arriving materialised the test fails loudly rather than going green for the wrong reason.

11/11 in SchemaUtilsTest, and the wider *schemas* / *RowTest* suites pass. spotlessCheck, checkstyleMain, checkstyleTest clean.

Row#toString throws for an ITERABLE field holding a plain Iterable:

  Schema s = Schema.builder().addStringField("k")
      .addIterableField("vals", FieldType.STRING).build();
  Row.withSchema(s).attachValues("k1", () -> list.iterator()).toString();

  IllegalArgumentException: value type is '...' for field type 'ITERABLE'

toPrettyFieldValueString demanded a List before iterating. An ITERABLE
field declares an Iterable, so the guard is stricter than the type it
guards, and the branch only ever iterates and counts -- both of which
Iterable supports once materialised.

Losing a whole Row's rendering is a poor trade for a stricter check
here, since this is what Row#toString calls: one unusual field takes out
logging and debugger output for every field beside it.

The guard now requires Iterable and materialises once for the size
count. A value that is neither still throws the same exception.

Two tests: a bare Iterable on an ITERABLE field, which fails on master,
and an ordinary List array as a control that passes either way. The
first also asserts the value really is not a List before rendering, so
it cannot go green for the wrong reason if the value ever starts
arriving materialised.
@github-actions github-actions Bot added the java label Aug 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@PDGGK

PDGGK commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

The one red check is beam_PreCommit_Java_IOs_Direct, and the failure is in a module this PR does not touch:

DebeziumReadSchemaTransformTest > testNoProblem[0] FAILED
    io.debezium.DebeziumException: Creation of replication slot failed
    Caused by:
    org.postgresql.util.PSQLException: ERROR: library "decoderbufs" may not be used as an output plugin
> Task :sdks:java:io:debezium:test FAILED
51 tests completed, 1 failed

That is the Postgres testcontainer refusing to load its logical-decoding output plugin, in sdks/java/io/debezium. This PR changes sdks/java/core/.../schemas/SchemaUtils.java and its test; 51 of 52 tests in that job passed and nothing in the failure touches schema rendering.

@PDGGK

PDGGK commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

assign set of reviewers

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).


@SuppressWarnings("unchecked")
List<Object> list = (List<Object>) value;
List<Object> list = Lists.newArrayList((Iterable<Object>) value);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: keep the original iterable instead of recreating the object as a list? use guava Iterables.isEmpty and Iterables.size for the lines below

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good instinct, and I've taken it — but not with Iterables.isEmpty / Iterables.size, because those would introduce a bug here. Details, since the reason is easy to miss:

As the block stood it touched the collection three times — isEmpty(), then size(), then the for. A List absorbs that for free, which is why it was fine before. An Iterable that can only be read once does not, and that is exactly the kind of value this PR widens support to. So the copy was load-bearing rather than gratuitous.

I added a test that hands out its iterator exactly once and throws on a second attempt. With your literal suggestion applied:

testToPrettyStringRendersAnIterableThatCanOnlyBeReadOnce
  java.lang.IllegalStateException: iterated more than once

1 of 13 failing, and it's that one.

What does satisfy the point is rendering in a single pass, which drops both the copy and the multi-traversal assumption:

StringBuilder sb = new StringBuilder();
sb.append("[\n");
boolean empty = true;
for (Object element : elements) {
  if (!empty) {
    sb.append(",\n");
  }
  sb.append(nextPrefix).append(toPrettyFieldValueString(elementType, element, nextPrefix));
  empty = false;
}
if (empty) {
  return "[]";
}
sb.append("\n").append(prefix).append("]");

Output is unchanged — same ",\n" separators, and an empty collection still renders as [], which now has its own test too. Lists is no longer imported. spotlessJavaCheck, checkstyleMain and checkstyleTest are clean, and SchemaUtilsTest plus RowTest pass.

Pushed in f297bcf. Thanks for the look.

Review feedback from @ahmedabu98: keep the original iterable rather than
recreating it as a list.

The copy was load-bearing as the block stood, because it asked the
collection for isEmpty(), then size(), then iterated it -- three
traversals. A List absorbs that for free; an Iterable that can only be
read once does not, and that is exactly the kind of value this PR widens
support to.

Doing it in a single pass gets both: no copy, and no assumption that the
collection can be traversed more than once. Output is unchanged --
elements are separated by ",\n" as before, and an empty collection still
renders as "[]".

Two tests added. The one-shot case hands out its iterator exactly once
and throws on a second attempt; with Iterables.isEmpty plus
Iterables.size over the raw iterable it fails with "iterated more than
once", and it is the only one of the thirteen that does.
@PDGGK

PDGGK commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Run Java_IOs_Direct PreCommit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Row.toString throws for an ITERABLE field that is not backed by a List

2 participants