What is the problem the feature request solves?
shuffle_bench builds its input with SessionContext::read_parquet under a default SessionConfig, and DataFusion's schema_force_view_types defaults to true, so string and binary columns reach ShuffleWriterExec as Utf8View and BinaryView. Comet never hands the shuffle writer those types: the serde maps Spark String to Utf8, and planner.rs casts UDF results back from Utf8View to Utf8 with the comment that Comet does not yet support view types. The bench is measuring the writer on a data shape production does not produce.
That would be a minor fidelity gap except that view arrays are the case the writer handles worst. BufBatchWriter configures its BatchCoalescer with with_biggest_coalesce_batch_size(Some(batch_size - 1)), so any produced batch of at least batch_size rows bypasses the coalescer and is serialized verbatim. An interleaved view array keeps the backing data buffers of every input batch it drew rows from, and IPC writes all of them in full, so the block carries far more bytes than its own rows.
Measured on a 5 column input with 2 string columns, 4M rows, 200 partitions, no compression, comparing the output file against the writer's own data size metric:
| batch size |
output file |
data size |
ratio |
| 2048 |
33.17 GiB |
236.95 MiB |
~143x |
| 4096 |
14.86 GiB |
242.07 MiB |
~63x |
| 8192 |
7.62 GiB |
242.33 MiB |
~32x |
| 16384 |
4.00 GiB |
243.96 MiB |
~17x |
| 20480 |
294.92 MiB |
245.45 MiB |
~1.2x |
The cliff at 20480 is where a partition's 20000 rows stop exceeding the threshold and go through the coalescer, which compacts the views. The same configuration on a numeric-only input writes 62.81 MiB for 62.00 MiB of data, and lowering --max-buffer-bytes shrinks the ratio in step with the buffered volume each block interleaves from.
The practical effect is that any string-heavy shuffle_bench number is dominated by amplification real Comet does not have, with encode time and write time inflated in proportion. #5198 points at this bench for measuring its items 1 and 6.
Describe the potential solution
Set datafusion.execution.parquet.schema_force_view_types to false in the bench's SessionConfig so the input matches what Comet's planner produces.
Two smaller things are worth handling alongside it. The header line prints a schema read separately through ParquetRecordBatchReaderBuilder, which reports the file's Utf8 rather than the Utf8View that actually executes, so the mismatch is invisible in the output. And the amplification itself is worth remembering if Comet ever does take on view types, since neither the local writer nor the RSS writer's compact_array compacts them today.
Additional context
No response
What is the problem the feature request solves?
shuffle_benchbuilds its input withSessionContext::read_parquetunder a defaultSessionConfig, and DataFusion'sschema_force_view_typesdefaults to true, so string and binary columns reachShuffleWriterExecasUtf8ViewandBinaryView. Comet never hands the shuffle writer those types: the serde maps SparkStringtoUtf8, andplanner.rscasts UDF results back fromUtf8ViewtoUtf8with the comment that Comet does not yet support view types. The bench is measuring the writer on a data shape production does not produce.That would be a minor fidelity gap except that view arrays are the case the writer handles worst.
BufBatchWriterconfigures itsBatchCoalescerwithwith_biggest_coalesce_batch_size(Some(batch_size - 1)), so any produced batch of at leastbatch_sizerows bypasses the coalescer and is serialized verbatim. An interleaved view array keeps the backing data buffers of every input batch it drew rows from, and IPC writes all of them in full, so the block carries far more bytes than its own rows.Measured on a 5 column input with 2 string columns, 4M rows, 200 partitions, no compression, comparing the output file against the writer's own
data sizemetric:The cliff at 20480 is where a partition's 20000 rows stop exceeding the threshold and go through the coalescer, which compacts the views. The same configuration on a numeric-only input writes 62.81 MiB for 62.00 MiB of data, and lowering
--max-buffer-bytesshrinks the ratio in step with the buffered volume each block interleaves from.The practical effect is that any string-heavy
shuffle_benchnumber is dominated by amplification real Comet does not have, with encode time and write time inflated in proportion. #5198 points at this bench for measuring its items 1 and 6.Describe the potential solution
Set
datafusion.execution.parquet.schema_force_view_typesto false in the bench'sSessionConfigso the input matches what Comet's planner produces.Two smaller things are worth handling alongside it. The header line prints a schema read separately through
ParquetRecordBatchReaderBuilder, which reports the file'sUtf8rather than theUtf8Viewthat actually executes, so the mismatch is invisible in the output. And the amplification itself is worth remembering if Comet ever does take on view types, since neither the local writer nor the RSS writer'scompact_arraycompacts them today.Additional context
No response