pager-hydration: reproducible experiment for pager impact on hydration - #37721
pager-hydration: reproducible experiment for pager impact on hydration#37721antiguru wants to merge 2 commits into
Conversation
Design doc and driver for measuring the column-paged batcher's spill-to-disk
("pager") impact on hydration time versus the legacy batcher, across TPCH scale
factors on staging. Uses ticking sources for honest re-hydration timing and
per-replica flag overrides keyed by replica name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_local.py drives the experiment against a local bin/environmentd, flipping the pager flag globally and sequentially since scoped per-replica parameters are not reproducible locally. README documents both the staging (run.py) and local paths, the out-of-band flag setup, invocation, outputs, and Grafana series, so the experiment is reproducible. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resume config: clean-slate pager variant ladder (SF100)Notes to resume the pager hydration re-measurement in a fresh session. Why we are re-runningThe earlier pager numbers (~1.1–1.5×) were contaminated by a stale That run used a non-default mixture (spill on, lz4 on, pageout off, budget 0.85). This ladder restarts from the code defaults and turns on one layer at a time. Code defaults (from
|
| replica name | enable | spill | lz4 | pageout | isolates |
|---|---|---|---|---|---|
legacy |
— | — | — | — | baseline, all code defaults |
paged |
✓ | — | — | — | paged data structure, resident-only (no spill) — pure structure cost |
paged_spill |
✓ | ✓ | — | — | + pager eviction, uncompressed spill |
paged_lz4 |
✓ | ✓ | ✓ | — | + lz4 compression on spill |
paged_pageout |
✓ | ✓ | ✓ | ✓ | + eager MADV_PAGEOUT (full stack) |
Scoped override each replica-name segment sets (full set to reach the rung, since env-wide is all default-off):
legacy— none (inherits defaults)paged—enable_column_paged_batcher=truepaged_spill—enable_column_paged_batcher=true,enable_column_paged_batcher_spill=truepaged_lz4—enable_column_paged_batcher=true,enable_column_paged_batcher_spill=true,column_paged_batcher_lz4=truepaged_pageout—enable_column_paged_batcher=true,enable_column_paged_batcher_spill=true,column_paged_batcher_lz4=true,column_paged_batcher_swap_pageout=true
Leave column_paged_batcher_budget_fraction at the default 0.05 everywhere. On M.1-8xlarge that is ~23.5 GiB budget vs a ~543 GiB working set, so the spill rungs actually spill hard (the 0.85 run spilled nothing). That is what makes paged_spill / paged_lz4 / paged_pageout meaningful.
Setup for the fresh session
- Clear the current env-wide overrides for the five pager flags so they read as code defaults. They appear to be LD-managed (
ALTER SYSTEM RESETdid not stick for a sibling flag), so clear them in the flag system, then verify againstmz_internal.mz_overridden_system_parameters(expect no pager rows). - Set
REPLICAS = ["legacy","paged","paged_spill","paged_lz4","paged_pageout"]inmisc/experiments/pager-hydration/run.py. It only measures replicas it creates; the segment names must match. All five hydrate in parallel in the shared test cluster, one M.1-8xlarge each (5 machines concurrently, but fast). - Resume the source:
ALTER CLUSTER ldgen_sf100 SET (REPLICATION FACTOR 1)and wait for the source to reportrunning(run.py polls this). Re-pause to RF 0 when done. - Run:
python3 run.py --scale-factors 100 --trials 3 --size M.1-8xlarge --config <writable mz.toml> --mz-bin <built mz> --region aws/us-east-1 --profile staging --outdir <dir>. Resumable (completed cells skip).
Metrics per replica window (windows.csv gives replica id + start/end)
- Hydration time:
mz_internal.mz_compute_hydration_times.time_ns(+ wall-clock frommz_hydration_statuses). - Peak memory (RAM+swap):
max_over_time(mz_memory_limiter_memory_usage_bytes{cluster_environmentd_materialize_cloud_replica_id="<uNNN>"}[...]). - Managed spill volume:
mz_column_pager_paged_bytes_out_total{...}(expect 0 forlegacy/paged, >0 for the spill rungs at budget 0.05). - Prometheus datasource uid
Ks85Oh14z; namespaceenvironment-4bd00568-7de9-4df4-90b4-5af076164555-0.
Optional complement: a Polar Signals on-CPU profile of paged vs legacy during hydration pinpoints which batcher functions carry the extra CPU. On-CPU is the right tool here since the cost is not spill-wait.
Environment note: staging is on clean upstream/main (build 7f6c52776, runtime-identical to the doc-only #37717 ff2ddcc36), backpressure disabled, no instrumentation.
Ladder run: first results + on-CPU root cause (SF100, M.1-8xlarge)First trial of the variant ladder is in ( The entire overhead is the paged data structure. Spill and lz4 add nothing.Geomean hydration time vs
The ~1.85× lands entirely on the Root cause (Polar Signals, on-CPU, paged vs legacy, same hydration window)The column pager allocates each chunk as a large
Actionable lever: cut Caveats and next steps
|
Consolidated results: pressure is the whole story (SF100, three machine sizes)Ran the four-rung ladder at three replica sizes to sweep memory pressure, holding the working set (~543 GiB) fixed and shrinking RAM. Correction to the previous comment: that used the trial-1 (n=1) number for Hydration time vs
|
| box | RAM / working set | paged | paged_spill | paged_lz4 | n |
|---|---|---|---|---|---|
| M.1-8xlarge (470 GiB) | ~1.15x | 2.12x | 2.13x | 2.15x | 3 trials |
| M.1-3xlarge (182 GiB) | ~3x | 1.23x | 1.28x | 1.29x | 3 trials |
| M.1-2xlarge (121 GiB) | ~4.5x | 1.05x (median 0.88x) | 1.15x | 1.19x | 1 trial |
rehydrate scenario (same trend, slightly lower):
| box | paged | paged_spill | paged_lz4 |
|---|---|---|---|
| M.1-8xlarge | 1.84x | 1.87x | 1.88x |
| M.1-3xlarge | 1.17x | 1.23x | 1.21x |
The paged batcher's relative penalty collapses as memory pressure rises: 2.12x when the data fits RAM, 1.23x at 3x RAM, and 1.05x (median 0.88x, i.e. faster than legacy) at 4.5x RAM. On the big box legacy runs mostly resident and paged's fixed allocation overhead dominates. On the small boxes legacy has no managed spill, thrashes kernel swap, and slows down enough that the pager reaches parity. Legacy did not OOM at either smaller size.
Managed spill fired hard and is cheap. lz4 buys ~41% for ~1%.
Per-replica mz_column_pager_paged_bytes_out_total over the hydration window (trial-1 sample):
| box | legacy | paged | paged_spill | paged_lz4 |
|---|---|---|---|---|
| M.1-3xlarge | 0 | 0 | 603 GB | 358 GB |
| M.1-2xlarge | 0 | 0 | 1.84 TB | 1.08 TB |
legacy and paged (resident) do zero managed spill, as designed.
paged_spill spilled up to 1.84 TB, yet costs only ~5-10% over paged in time, so managed spill is close to free.
paged_lz4 spills ~41% fewer bytes (358 vs 603 GB, 1.08 vs 1.84 TB) at ~1% extra time, so lz4 is a large footprint win for almost nothing.
Where the fixed overhead comes from (on-CPU, paged vs legacy)
The 2.12x on the big box is jemalloc large-extent churn. ColumnPager::page allocates each chunk as a large RawVec, so every chunk round-trips the large-object extent allocator; legacy uses a columnation arena (bump-copy, no per-chunk allocator traffic).
ColumnPager::page<((Row,Row),Timestamp,Diff)> ~1m06.5s cumulative
-> RawVecInner::try_allocate_in -> _rjem_je_large_palloc
-> arena_extent_alloc_large -> pa_alloc -> ecache_alloc -> extent_recycle ~71s
extent_recycle + extent_split_impl + extent_try_coalesce is ~71-73s on paged and ~0 on legacy.
Lever: cut ColumnPager::page's large-allocation frequency (chunk-buffer pooling, larger chunks, or arena backing).
The existing column_paged_batcher_use_pool flag (off) is the likely mitigation and is worth its own rung.
Caveats
- M.1-2xlarge is n=1: staging JWKS auth blips ("Error fetching JWKS") aborted trials 2-3. The trend is monotonic and consistent, but the 4.5x row wants a repeat.
paged_pageoutrung omitted:column_paged_batcher_swap_pageoutis notReplica-scoped in the deployed binary, so it cannot bind a per-replica override (CPU-177).
New pager (build
|
| box | regime | rung | old pager | new pager |
|---|---|---|---|---|
| M.1-8xlarge | resident (~1.15x RAM) | paged | 2.12x | 0.95x |
| M.1-8xlarge | resident | paged_spill | 2.13x | 1.47x |
| M.1-3xlarge | deep swap (~3x RAM) | paged | 1.23x | 1.12x |
| M.1-3xlarge | deep swap | paged_spill | 1.28x | 1.24x |
rehydrate tracks initial: 8xlarge paged 1.84x -> 1.05x, 3xlarge paged 1.17x -> 1.13x.
The resident-regime paged rung went from 2.12x (worse than legacy) to 0.95x (faster than legacy). The buffer pool eliminated the per-chunk large-allocation traffic that dominated the old profile, so the columnar-native batcher no longer pays an allocator tax to build arrangements.
Managed spill
Spill volume via the new pool metrics (mz_column_pool_extent_bytes_written_total, trial-1):
| box | paged (no managed spill) | paged_spill |
|---|---|---|
| M.1-8xlarge | 0 | 299 GB |
| M.1-3xlarge | 0 | 752 GB |
paged_spill costs more than paged when RAM nearly suffices (8xlarge 1.47x vs 0.95x): spilling ~299 GB has a price when the working set is only slightly over RAM. Under real pressure (3xlarge) the gap narrows (1.24x vs 1.12x). The pool also holds a compressed-but-resident tier (peaked ~38 GB on 3xlarge, under the pool_rss_target_fraction = 0.25 ceiling) as headroom above the slot budget before paging to the device.
eager_backing is a no-win for hydration
Swept eager_backing off vs on, env-wide, on M.1-3xlarge:
| rung | eager off | eager on |
|---|---|---|
| paged | 1.12x | 1.16x |
| paged_spill | 1.24x | 1.27x |
Eager write-behind fired (~185k eager-backs) but slightly increased hydration time and did not grow the resident tier (~38 GB either way). Hydration pressure is one-shot and chunks are short-lived, so pre-compressing them is wasted CPU. Keep eager_backing off for hydration-bound workloads.
Open
paged is now at parity-to-slightly-faster than legacy, not a decisive win. The allocator tax is gone, so what remains is the columnar batcher's own build path vs columnation's arena. Profiling that next.
Notes
- n=3 trials on both boxes, both scenarios.
pageoutrung dropped (flag removed in this build;lz4/swap_pageoutreplaced by the pool +eager_backing+pool_rss_target_fraction). spill_worker_countwas pinned to 2 for the experiment env (the env fallthrough resolved 0, which evicts inline and convoys workers).
Re-run on the rebased stack: the pooled batcher is at parity, and the spill cost is a thread-count convoyRe-ran the three-rung ladder ( Hydration time vs
|
| box | regime | rung | initial | rehydrate | previous round |
|---|---|---|---|---|---|
| M.1-8xlarge | resident, ~1.15x RAM | paged |
1.03x | 1.01x | 0.95x |
| M.1-8xlarge | resident | paged_spill |
1.54x | 1.61x | 1.47x |
| M.1-3xlarge | deep swap, ~3x RAM | paged |
1.14x | 1.15x | 1.12x |
| M.1-3xlarge | deep swap | paged_spill |
1.34x | 1.40x | 1.24x |
Every cell reproduces the previous round within single-trial noise. column_chunk_compress_min_depth (new, default 1) and the whole-chunk extract pass-through did not move this workload, and the pressure gradient still holds: the pooled batcher's relative cost shrinks as the box shrinks, because legacy's kernel-swap thrash degrades faster than the pool's overhead does.
The headline is unchanged and worth stating plainly. paged is at parity with legacy, not ahead of it.
Why paged only ties legacy
Closing the open item from the previous comment. On-CPU profiles of paged against legacy on the resident box, taken on the earlier pooled build, show the old jemalloc large-extent churn gone: extent_recycle fell from 71s to 1.9s. What replaces it, and cancels the columnar win:
CompactBytes::extend_from_slice, 7.4s flat. Every row is byte-copied into a growable chunk buffer, where columnation bump-appends into an arena for approximately nothing. This is the dominant residual.- Eager
madvise/do_madvise, about 16s against roughly 3s of lazy reclaim on legacy. ColumnChunk::settle, 4.5s against 2.3s.tcache_alloc_small_hard, 4.9s. The allocation pressure moved from large to small.
The reading is that the pressure moved rather than disappeared: columns on the wire hit the roughly 2 MiB ColumnBuilder ship threshold, so the chunk buffers themselves become repeated 2 MiB allocations. The candidate fix is to source those buffers from the pool instead of the allocator, and the clearest single lever is the per-row byte copy.
The paged_spill cost is a convoy on the compression threads
Instrumented the resident-box window with node and container metrics. The spill rung is not disk-bound and not compute-bound.
- CPU totals over the shared window are within 1% across all three rungs (
legacy19746u/5279s,paged19757u/5159s,paged_spill20880u/4311s) whilepaged_spilltakes 1.5x longer to hydrate. The extra time is off-CPU. legacyandpagedramp to 0.96 and 0.98 of the 62-core box, with 23% to 25% of the whole box in kernel, which is the fault and swap machinery.paged_spillnever exceeds 0.66 and is still working when the other two have finished.- Device traffic favors the spill rung: writes and reads of 238/57 GB against
legacy290/65 GB andpaged324/84 GB. Device busy peaks at 1.0 forpaged_spillagainst 2.6 and 2.9. Bandwidth is not the wall. mz_column_pool_spill_in_flightsits pinned at 62, exactly the worker count and one entry per worker, for about seven minutes, then drops to zero.
The arithmetic matches a service-rate ceiling. 314,950 compressing evictions over roughly 660s is 477 per second, or 239 per thread per second at about 950 KB compressed each, which is close to two cores of lz4 and to the observed sustained 600 MB/s of extent writes.
An A/B confirms it. Raising column_paged_batcher_spill_worker_count from 2 to 8 on the experiment environment, changing nothing else:
| rung | 2 threads | 8 threads |
|---|---|---|
paged initial |
1.03x | 1.05x |
paged rehydrate |
1.01x | 1.03x |
paged_spill initial |
1.54x | 1.22x |
paged_spill rehydrate |
1.61x | 1.24x |
The paged rung is untouched, as expected, since without the spill gate it never installs the pool. The ladder's own window also fell from 670s to 542s, because paged_spill was the straggler. With 8 threads the rung spills more (434 GB over 458k evictions, against 298 GB over 315k) and still finishes sooner, since draining faster frees budget faster.
So the default of 2 is mis-sized for hydration on a wide replica. It is a fixed service rate facing a producer count that scales with workers, so the convoy worsens as replicas get wider. Scaling the thread count with worker count is the obvious change to consider.
Other observations
evictions_cheapis 0 in every run. Witheager_backingoff nothing is pre-backed, so every eviction pays compression inline of the budget.admissions_budget,admissions_stealandadmissions_deniedare all 0, so there is no re-admission thrash.writes_elidedreaches 3.07M, so most chunks die before their backing write completes.- Peak working set is identical across the three rungs at roughly 470 GiB, the box's RAM. The pool buys no peak-memory headroom here.
- Managed spill defers kernel swap rather than removing it: 0 GiB until late in the window, then 87 GiB, against 130 GiB for
legacyand 142 GiB forpaged. Major faults per second peak at 542 forlegacy, 113k forpaged, and 13.8k forpaged_spill.
Where main stands
Worth recording, because it is easy to get wrong when picking a build to measure. main carries the pool (#37718), its configuration and metrics (#37719), the budget fix (#37824), UnloadChunk (#37954) and ColumnChunk (#37955), but not the compute arrange adoption and not the pager deletion. On main the enable_column_paged_batcher gate still routes Col2ValPagedBatcher, which is ColumnMergeBatcher over column_pager::PagedColumn, the pre-pool pager. Measuring the pooled path needs a build from this stack.
Notes
- One trial per cell, so treat single cells as directional. The previous round's 8xlarge reading moved by 0.27x between one trial and three.
compute_hydration_concurrencyis 1 in the experiment environment, as it was for the earlier rounds, so the comparison holds.- The backpressure override stays unset, so the read path is not the confound it once was.
column_paged_batcher_spill_worker_counthas been restored to 2, the sources are back at replication factor 0, and the test clusters are torn down.- The runner now carries
analyze.py, which computes these ratios. It reproduces the previously published figures from the stored CSV of the earlier round. Per-objecthydration_secondsmeasures elapsed time inside a shared parallel window, so only per-object ratios aggregated geometrically are meaningful, and an index's owntime_nsis preferred where it exists.
Hydration time budget: the out-of-core axis tops out near 1.6x, and we have already collected most of itStepping back from tuning knobs. The goal for this work was roughly a 2x hydration improvement, and the best measured result so far is about 5% against the legacy batcher. This comment decomposes where hydration time actually goes, so each lever can be given a ceiling rather than another sweep. All numbers are SF100 on M.1-8xlarge (62 workers, 470 GiB), build MethodFor each rung, its wall-clock span is the maximum per-object elapsed time in the measurement window, and its CPU is the process rusage counters
The budget,
|
| config | span | utime | stime | stalled |
|---|---|---|---|---|
legacy |
486s | 63% | 25% | 12% |
paged |
606s | 46% | 23% | 31% |
paged_spill, 2 spill threads |
775s | 38% | 12% | 50% |
paged_spill, 128 spill threads |
465s | 80% | 17% | 2% |
The spill work tunes the stalled bucket. In the best configuration that bucket is already down to 2%, so there is essentially nothing left to recover there.
Composed ceilings
Starting from the legacy baseline of 486s and removing each cost entirely:
| lever | implied floor | cumulative gain |
|---|---|---|
| remove every stall, same CPU work | 428s | 1.14x |
| also remove all kernel and paging time | 306s | 1.59x |
| run at the lowest utime any rung achieved (17,267 core-seconds) | 278s | 1.75x |
The second row is the one that matters. Legacy is 63% utime, so removing the other 37% entirely, meaning every stall and every kernel cycle, leaves 486s x 0.63 = 306s. That is 1.59x, and it is the ceiling of the whole out-of-core axis.
2x is 243s, which across 62 cores is a total budget of 15,066 core-seconds. Legacy's utime alone is 19,092. So 2x requires three things at once: eliminate all paging, eliminate all stalls, and cut application CPU by a further 21%. Only the first two are on this axis.
Two refinements, both in the pessimistic direction:
- Not all of the 25% stime is reclaimable. It mixes paging with the syscalls hydration needs to read the snapshot from persist at all. Paging is the bulk of it on this workload, since legacy's kernel profile is dominated by
do_translation_fault(1m22s),handle_mm_fault(1m12s) anddo_try_to_free_pages(1m9s), but the floor is above zero. - Dividing utime by 62 assumes the application work spreads perfectly across all workers with no dependency serialization. The 2% stall measured on
paged_spillat 128 threads says that holds on this workload, but it is an assumption, and it is the first one to break on a skewed or dependency-heavy dataflow.
Where the remaining time is
In the best configuration 80% of the budget is utime. Ranked hot paths from the on-CPU profile of the compute worker threads: the timely exchange push path, consolidation sorts, Row and CompactBytes copies, parquet decode, decNumber decimal handling, and lz4 encode on the spill rung. Those are the only levers with 2x-scale headroom.
Spill worker sweep, for the record
Sweeping column_paged_batcher_spill_worker_count on the same shape, paged_spill versus legacy:
| threads | workers per thread | initial | large objects | CPU floor | collapse |
|---|---|---|---|---|---|
| 2 | 31:1 | 1.57x | 1.77x | 1.8 cores | ~4 min |
| 4 | 15.5:1 | 1.54x | 1.70x | 2.9 cores | ~2 min |
| 8 | 7.8:1 | 1.12x | 1.08x | 3.3 cores | ~1 min |
| 16 | 3.9:1 | 1.22x | 1.27x | not sampled | not sampled |
| 128 | 0.5:1 | 1.03x | 0.95x | none | none |
The rehydrate scenario tracks it: 1.47x, 1.42x, 1.22x, 1.06x, 1.10x for the same ladder.
The inflection is sharp and sits between 4 and 8 threads. Above 8 the curve is flat within single-trial noise, with 8, 16 and 128 all landing in a 1.03x to 1.22x band across both scenarios. Two threads is simply under-provisioned rather than there being a useful workers-per-thread ratio to tune toward. The mechanism is visible directly in pod CPU: with 2 threads the replica sits at the spill-thread count for minutes at a time, because every worker blocks in ColumnChunk::settle waiting for pool admission and only the spill threads keep running. At 128 threads it never drops below about 45 cores. Turning compression off at 2 threads gives 1.29x, so lz4 is roughly half the per-eviction cost, but threads are the stronger lever: 128 threads with compression beats 2 threads without it.
Also worth recording, since it reconciles this experiment with the upsert campaign's TPCH ladder: at 8 threads and above the per-object size gradient inverts. Large objects (legacy over 100s) go from 1.77x to 0.95x and start winning, while small objects settle at a flat tax around 1.14x. That is the same signature the campaign reported on a 200cc replica, which runs about 6 workers and therefore never had a convoy at 2 threads. The whole discrepancy between the two reports was workers per spill thread.
Two limits on this analysis
The profiling backend here exposes only parca_agent:samples:count:cpu:nanoseconds:delta and memory:inuse_space. There is no off-CPU or wallclock profile type, so the stalled bucket can only be bounded by subtraction, not attributed by stack. Separately, the flamegraph root cumulative is not additive across series (two thread filters returned the same total as the unfiltered pod), so hot functions can be ranked but not converted into a reliable percentage split of utime.
Suggested next step
Get a trustworthy utime breakdown for a single legacy hydration of the largest object, split into decode, exchange, sort and consolidate, and copy. That is cheap next to another staging sweep and it decides the question this budget raises: whether a 35% cut in application CPU is plausible, or whether the hydration target should be restated at something like 1.5x from the pool and paging work and the remaining effort redirected at the dataflow itself.
The same budget in absolute terms: the pool trades kernel work for user work at an unfavourable rateThe percentages in the previous comment have different denominators, since each rung has its own span. Redone in core-seconds, against
The pool removed 5,308 core-seconds of paging and stall, and spent 4,006 of them back on application CPU. The net is 1,302 core-seconds, which at 62 cores is exactly the 21 seconds of wall clock observed (486s to 465s). The apparent move from 63% to 80% utime is mostly the denominator shrinking plus utime growing, not work disappearing. This raises the floor rather than lowering it
The spill rung is far more efficient against its own ceiling, 125% of floor versus 158%, but its ceiling is 65s worse. A perfected The added utime scales with bytes spilledWithin-run deltas of
Both lz4 configurations land near 4 to 5 core-seconds of utime per GB of extents written. That produces a tension worth naming: raising the thread count doubled the spill volume, 389 GB to 771 GB, because draining faster frees budget faster and provokes more eviction churn. The change that removed the stall also inflated the floor. Going from 2 to 128 threads cut the span by 310s and raised the floor by 75s, from 298s to 373s. Turning the codec off at the same thread count moved the delta from +1,710 to +1,380, roughly 20%, so compression is the minority of the pool's added CPU. The remaining 80% is pool bookkeeping and copying, the Caveat on those two small numbers. Effect on the floor analysisThe 1.59x ceiling is unchanged, since it rests on |
Managed spill has a crossover at ~4.5x working set to RAM, and every earlier number in this thread was confounded by hydration concurrencyTwo corrections to what I posted above, both of which change the conclusion rather than refine it. The 1.59x ceiling was a property of the box, not of hydrationThe time budget I published earlier (63% utime / 25% stime / 12% stalled, ceiling 1.59x) was measured only on M.1-8xlarge, where the working set is barely above RAM. The split depends entirely on memory pressure:
On a large box paging shows up as
|
| box | legacy hc=4 | legacy hc=1 | spill hc=4 | spill hc=1 |
|---|---|---|---|---|
| M.1-3xlarge | 2867.6s | 1378.5s | 1552.7s | 1513.5s |
| M.1-2xlarge | 5747.2s | 3617.4s | 2615.5s | 2679.7s |
| M.1-1.5xlarge | 7801.5s | 6190.2s | 4305.5s | 4287.8s |
Spill moves 0.4% to 2.6% across the two settings. Legacy moves 1.26x to 2.08x. Every hc=4 comparison therefore mixes the pager's effect with legacy's concurrency penalty, and at 2.98x pressure the apparent 1.85x win was entirely the latter: at hc=1 legacy is the faster rung there.
This also retires the one capability claim. On 1600cc at hc=4 legacy failed to hydrate 14 of 28 objects inside 12000s while every spill rung finished in about 1120s. At hc=1 legacy completed all 28.
The crossover, measured
Eleven cells, all at hc=1, legacy against spill with 8 spill threads, initial scenario, one trial each. The x-axis is measured pressure, peak container_memory_swap plus peak container_memory_working_set_bytes on the legacy rung divided by RAM.
| pressure | SF | size | span legacy/spill | per-object geomean | sum time_ns |
winner |
|---|---|---|---|---|---|---|
| 2.31x | 10 | M.1-small | 0.83 | 1.18 | 1.25 | legacy |
| 2.87x | 200 | M.1-8xlarge | 0.72 | 1.31 | 1.30 | legacy |
| 3.39x | 30 | M.1-large | 0.89 | 1.15 | 1.15 | legacy |
| 4.28x | 30 | M.1-medium | 0.97 | 1.10 | 1.05 | legacy |
| 4.49x | 10 | M.1-xsmall | 0.88 | 1.23 | 1.15 | legacy |
| 4.67x | 100 | M.1-2xlarge | 1.35 | 0.92 | 0.77 | spill |
| 4.74x | 200 | M.1-4xlarge | 1.27 | 0.97 | 0.82 | spill |
| 5.40x | 30 | M.1-small | 1.36 | 0.89 | 0.76 | spill |
| 6.02x | 100 | M.1-1.5xlarge | 1.44 | 0.79 | 0.69 | spill |
| 6.04x | 200 | M.1-3xlarge | 1.58 | 0.77 | 0.63 | spill |
| 6.95x | 10 | M.1-micro | 1.33 | 0.80 | 0.52 | spill, both incomplete |
All three metrics flip between 4.49x and 4.67x. Points from a 20x range of scale factor interleave monotonically instead of forming four separate curves, so pressure is the axis rather than data volume. The sharpest check is the 6.02x and 6.04x pair, SF100 on a 91 GiB box against SF200 on a 182 GiB box: span 1.44 against 1.58, geomean 0.79 against 0.77, summed time_ns 0.69 against 0.63.
Why the crossover exists
The pool adds application CPU everywhere: 21% on the 8xlarge, 33% on the 3xlarge, 32% on the 2xlarge. That raises its own floor, so below the crossover the added CPU is not repaid. Above it the stall bucket is large enough to pay for it. The pool buys no capacity: peak swap is within 2.5% across rungs on any box with headroom, so it changes which pages leave RAM and when, not how many.
Practical consequences
column_paged_batcher_spill_worker_countdefaults to 2, and that is the single worst setting measured. On the 8xlarge it made spill 1.54x slower than legacy. The knee is near 8 workers per spill thread, and 8, 16 and 48 threads are flat within noise at 62, 24 and 16 workers. Scale it with worker count, something likemax(2, workers/4).- Enable spill above roughly 4.5x working set to RAM, and leave it off below that, where it is a regression.
column_paged_batcher_lz4andcolumn_paged_batcher_swap_pageoutare still present as environment overrides but were deleted from the code on this build. Compression is controlled bycolumn_chunk_compress_min_depth, default 1, so it is on.
Caveats
Every cell is one trial, and legacy's own utime varied 14% across repeated runs earlier in this thread. The three SF30 cells overlapped an SF200 ingest and sit slightly closer to parity than their neighbours, so they are the ones to repeat first. Working set cannot be predicted from scale factor: it is not linear in SF, and the same workload wants less memory on a smaller box because the batcher budget scales with physical RAM, so the axis has to be measured rather than computed. M.1-micro is the only genuine capacity failure found, and both rungs failed there, legacy at 21 of 28 objects and spill at 25.
Disk ratio does nothing, the crossover is parity at 4.5x rather than a win, and the threshold depends on hydration concurrencyFollow-up to the pressure sweep above. Three results, one of which walks back a number I posted there. More disk buys the pager nothingSeven new cells at hc=1, legacy against spill with 8 spill threads, Row B, 2 cpu / 15.16 GiB, SF10, about 4.49x measured pressure:
Flat across a 16x span of disk allocation and three instance families. Absolute times drift slightly worse with more disk, which is plausibly the storage-optimised families trading per-core performance for NVMe, but at one trial per cell that is not worth asserting. The 100cc cell was a crash loop rather than a slow run. Its heap limit is 15.16 + 22.7 = 38 GiB and both rungs peaked at 41 to 42 GiB, so the memory limiter terminated and recreated them about twenty times each and hydration restarted every time. Its spans are meaningless. The 4.49x point is parity, not a winAdding the two new cells at that pressure to the two we already had gives four independent measurements:
They straddle 1.0 and average about 0.97. So 4.5x is where the two rungs draw level, and the crossover interval of 4.49x to 4.67x that I quoted above is narrower than the measurement noise. M.1-xsmall repeated at identical settings gave 0.88 and then 0.98, an 11% swing on the ratio, and the four cells at 4.49x span 17%. What survives is the ordering, which is clean across a 20x range of scale factor and three instance families. Useful margin starts at 4.67x and grows monotonically to 1.58x at 6x, so a rule with headroom above the noise is 5x, not 4.5x. At the shipped default the crossover is much lowerEverything above is at
Spill is concurrency-invariant within 3% on every box measured, legacy degrades 1.26x to 2.08x going from hc=1 to hc=4. So the threshold is a property of the pair, not of the pager: at hc=4 the crossover sits somewhere between 1.16x and 2.98x, a gap we never measured. That makes managed spill and a lower hydration concurrency two alternative fixes for overlapping ground. Keeping hc=4 means the pager pays off from roughly 2x pressure and an "enable above 5x" rule leaves wins on the table. Lowering concurrency on pressured replicas recovers most of legacy's loss for one dyncfg, and then the pager only earns its keep above 5x. Two cells in the 1.16x to 2.98x band at hc=4 would settle which. One outlier excluded, and why1600cc at about 3.0x pressure showed legacy 2.0x slower than its controls at identical cpu, memory and worker count (2106s against 1054s on M.1-4xlarge and 930s on D.1-4xlarge), moving 1020 GB of block I/O against 517 and 523 GB. Peak throughput was the highest of the six cells at 1062 MB/s, so the device was not the constraint, and both rungs averaged about the same sustained rate, so it moved twice the bytes for the same swap residency. I no longer think this is a disk effect. Row B says disk ratio does nothing, and the limiter demonstrably works: 100cc restarted 21 times at an 11% overshoot. Yet 1600cc peaked at roughly 724 GiB against a 587.5 GiB heap limit, a 23% overshoot, with zero restarts. What would improve the numbersNot another replica size. Exporting the node's |
1 participant
Captures the pager hydration experiment so it is reproducible.
Measures the impact of the column-paged batcher's spill-to-disk mechanism (the "pager") on hydration time, comparing a paged-batcher replica against the legacy columnation batcher across TPCH scale factors.
Contents:
doc/developer/design/20260716_pager_hydration_experiment.md— design and rationale.misc/experiments/pager-hydration/run.py— driver for a staging or cloud region over pgwire; two replicas differing only by name, with the flag mapping applied out of band via scoped per-replica system parameters.misc/experiments/pager-hydration/run_local.py— variant for a localbin/environmentd, flipping the pager flag globally and sequentially since scoped parameters are not reproducible locally.misc/experiments/pager-hydration/README.md— prerequisites, invocation, outputs, and Grafana series for both runners.Draft: infra to reproduce the measurement, not a product change.
🤖 Generated with Claude Code