Repository navigation
Commit 91eebbf
GH-3967: Skip the page copy for decompressors with stable output
GH-3150 (#3801) copies every decompressed page in ColumnChunkPageReadStore,
because the BytesInputDecompressor contract allows a decompressor to reuse
its output buffer. On the default heap path that copy is redundant for
UNCOMPRESSED (NO_OP returns the page as read) and for ZSTD/LZ4_RAW
(HeapBytesDecompressor already decompresses into a fresh array).
Add a package-private CodecFactory.BytesDecompressor.returnsStableBytes(),
false by default and overridden by NO_OP and by HeapBytesDecompressor for
eagerly materialized codecs. The reader skips the copy only for those;
DirectCodecFactory and third-party decompressors are still copied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1 parent 567c0ec commit 91eebbf
3 files changed
Lines changed: 125 additions & 7 deletions
File tree
- parquet-hadoop/src
- main/java/org/apache/parquet/hadoop
- test/java/org/apache/parquet/hadoop
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
91 | 96 | | |
92 | 97 | | |
93 | 98 | | |
| |||
173 | 178 | | |
174 | 179 | | |
175 | 180 | | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
176 | 190 | | |
177 | 191 | | |
178 | 192 | | |
| |||
192 | 206 | | |
193 | 207 | | |
194 | 208 | | |
195 | | - | |
| 209 | + | |
196 | 210 | | |
197 | 211 | | |
198 | 212 | | |
| |||
514 | 528 | | |
515 | 529 | | |
516 | 530 | | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
517 | 540 | | |
518 | 541 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | | - | |
181 | | - | |
| 180 | + | |
182 | 181 | | |
183 | 182 | | |
184 | 183 | | |
| |||
258 | 257 | | |
259 | 258 | | |
260 | 259 | | |
261 | | - | |
| 260 | + | |
262 | 261 | | |
263 | 262 | | |
264 | 263 | | |
| |||
297 | 296 | | |
298 | 297 | | |
299 | 298 | | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
300 | 317 | | |
301 | 318 | | |
302 | 319 | | |
| |||
327 | 344 | | |
328 | 345 | | |
329 | 346 | | |
330 | | - | |
331 | | - | |
332 | | - | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
333 | 350 | | |
334 | 351 | | |
335 | 352 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| 37 | + | |
36 | 38 | | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
40 | 42 | | |
| 43 | + | |
41 | 44 | | |
42 | 45 | | |
| 46 | + | |
43 | 47 | | |
44 | 48 | | |
45 | 49 | | |
| |||
80 | 84 | | |
81 | 85 | | |
82 | 86 | | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
83 | 144 | | |
84 | 145 | | |
85 | 146 | | |
| |||
159 | 220 | | |
160 | 221 | | |
161 | 222 | | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
162 | 240 | | |
163 | 241 | | |
164 | 242 | | |
| |||
0 commit comments