Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 481 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 481 commits into
sbryngelson wants to merge 481 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 60.77% 61.99% +1.22%
==========================================
Files 83 93 +10
Lines 20872 25638 +4766
Branches 3101 4206 +1105
==========================================
+ Hits 12685 15895 +3210
- Misses 6121 6991 +870
- Partials 2066 2752 +686 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
…l at its own dt Build the coupling into a recursion out of inc.1's s_amr_advance_subtree: - s_amr_advance_subtree gains a dt_sub arg (top call passes amr_dt_fine=dt/2; the recursion halves), so per-level dt (L1 dt/2, L2 dt/4, ...) falls out of the recursion. - s_amr_gather_from_parent_field(pblk, qp): gather a level>=2 block's coarse patch from a SPECIFIC parent snapshot field (reuses the existing s_amr_copy_parent_patch GPU kernel); the lock-step s_amr_gather_from_parent now delegates to it. - s_amr_advance_children: after each parent substep (parent @ t_b in q_cons, t_a in q_cons_stor - both live for free, no new storage), for each level+1 child gather its two ghost-lerp sources from those snapshots, recurse at dt_sub/2, then fold back with s_amr_restrict_to_parent + per-substep Berger-Colella s_amr_reflux_to_parent(dt_sub). - Driver (m_time_steppers) skips level>=2 blocks in the top-level subcycle loop; they are advanced/restricted/refluxed inside the parent's recursion. m_amr_registers needs ZERO changes: the per-substep weights (freg 1/r*rk3_w, creg rk3_w from the MFlowCode#31 lock-step child-capture) already close conservation, and the capture/consume timing is self-consistent across the recursion. inc.2 adds NO new GPU device kernels. VALIDATED np=1: static 2-level + amr_subcycle=T (case D95E1B08). L2 genuinely active (2-level vs 1-level solution differs 1.66e-2 where the wave crosses the block); global drift 9.7345E-10 mass / 1.3777E-09 energy = IDENTICAL to the single-level subcycle baseline (L2<->L1 reflux is internal, adds zero global leak). Self-test exact (reflux 3.55e-15). All 10 existing AMR goldens byte-identical. CPU and GPU(V100/OpenACC) results byte-for-byte identical - no present-table/sync divergence. New golden D95E1B08 protects the recursion.
…l subcycle conservation s_amr_advance_children is called as s_amr_advance_children(amr_cur, ...), so its intent(in) dummy pslot is argument-associated with the module variable amr_cur. The in-loop s_amr_select_slot(kc) reassigns amr_cur and silently corrupted pslot, so the final restore left amr_cur on the child slot: the driver's L1->L0 restrict/reflux then operated on the child, the L1 fine solution was discarded, and apply_reflux_state used the child's registers -> conservation leak (up to ~1e-3 with a moving L2; masked in the static golden where the child sits in smooth flow). Copy pslot to a local read before any slot switch. Regenerate D95E1B08 (captured the buggy state) and add 00E15144 (dynamic regrid + subcycle + multi-level). CPU+GPU(V100) both conserve to machine zero (~4e-15); full AMR golden group 47/47.
… refinement towers) Multi-level (amr_max_level>1) was hard-gated to num_procs=1. Enable it for a STATIC hierarchy (amr_regrid_int=0) at np>=2 via co-located refinement towers: the L2 child inherits its parent's rank (s_amr_test_multilevel already sets amr_block_owner(L2)=amr_block_owner(1)), so the L1<->L2 fold stays LOCAL (bit-identical to np=1) and only L0<->L1 crosses ranks via the existing Body-1 P2P. Changes: (1) checker gate now only prohibits DYNAMIC (amr_regrid_int>0) multi-level at np>1 - the cross-rank sensor-on-fine nesting is still np=1; (2) un-gate s_amr_test_multilevel for np>=2; (3) owner-guard s_amr_restrict_to_parent + s_amr_reflux_to_parent - without them the lock-step L2->parent fold dereferences a non-owner's unallocated parent slot (SIGSEGV on rank 1); the subcycle path already guarded. Validated: static 2-level conserves machine-zero at np=1/2/4, both lock-step and subcycle; the 4 np=1 multi-level goldens are unchanged (guards are no-ops at np=1). New golden 4644A339 (multi-level static np=2). GPU np=1 unchanged (no-op guards); np>=2 GPU validation is the multi-GPU scaling study (sbatch).
…sting + co-located owners)
… fine-block migration segfaults; use n>0 not num_dims which is unset at checker time)
…r arg + fine-fine halo buffer sizing on non-owner ranks)
…(missing per-substep fine-fine seam halo, not reflux/regrid ordering)
… subcycle driver for per-substep fine-fine seam halo)
…el-aware fine-fine halo extent + reflux seam-face exclusion + enable L2 tiling)
…zero + static under-refine) 1) s_amr_tile_box: clamp the per-dim tile size to >= 1. The lock-step multi-level nesting path passes amr_maxc_fit/2, which is 0 when a rank's fine half-extent is 1 (small subdomain at high np) -> integer div-by-zero (SIGFPE) instead of refining. 2) static multi-level (amr_max_level>1, amr_regrid_int=0) built its only L2 in s_amr_test_multilevel, which SILENTLY returned (no L2, under-resolved 'success') when amr_max_blocks was too small. Abort at the point of failure (the L1 tile count is only known there) + a checker gate for the trivially-detectable <2 case.
- s_amr_fine_slice now moves only the buff_size-deep near-seam slab it packs/ unpacks device<->host (interior transverse only), replacing the two full-block q_cons GPU_UPDATE loops in s_amr_fine_fine_halo. Byte-identical (same cells, same order) at a fraction of the PCIe volume - the halo runs per stage, 6x per fine step after the MFlowCode#28 transpose. - s_amr_reflux_to_parent: swap the sibling-seam detection nest (block outer, dim inner) so f_amr_parent_block - an O(nblocks) scan - is evaluated once per sibling instead of once per (sibling, dim). Behavior-identical. Both from the high-effort branch review. GPU (V100): all L1/L2-tiling np=2 goldens byte-identical (EF58E377 tiled-L2, F57C3A5B, 244B1E42, ADA042A2, 5EFB3277, ...).
…construction s_amr_test_multilevel was the ONLY builder of the static (amr_regrid_int=0) level-2 hierarchy, but it also ran ~90 lines of intrusive development checks on every populate: block-1 full save/restore, restrict-prolong / restrict-to-parent / reflux-to-parent identity probes (perturbing creg/freg), and three ES12.4 prints PER STEP. Those identities are now permanently protected by the static multi-level goldens (75AD6885, 4644A339, ADA042A2, D95E1B08) and the runtime conservation-defect probe, so the scaffolding is dead weight (and self-test spam) in production. Remove the checks, keep only the construction (inset -> L2 metadata -> reconcile -> fine geometry -> prolong -> device push -> persist), and rename the routine to s_amr_build_static_multilevel to match its now-sole purpose. The checks restored block 1 exactly and registers are re-zeroed each step, so output is byte-identical: GPU (V100) static ML goldens all pass unchanged (prints, which no golden captures, are simply gone).
s_amr_apply_reflux_state (L0/L1, coarse/sidx frame, unit weights from ownership, ref_ratio=2) and s_amr_reflux_apply_parent (L2->L1, parent-fine frame, sibling-seam weights, ref_ratio=slot) were near-verbatim copies of the same three-face flux correction, so a future physics/stencil fix could silently drift the two apart and break conservation on one path only (the review's concern). Extract the shared math into s_amr_reflux_apply_faces(q, islot, rr, dtl, olo, ohi, glo, ghi, woff, w_lo, w_hi, mlo, mhi): the caller passes all framing (outside index, creg-local loop range, transverse write origin, per-face weights, cell widths). Both callers become thin wrappers; the L2->L1 copy is deleted (net -60 lines). A zero weight now SKIPS the write (not multiply-by-0) so an unowned coarse face's out-of-bounds outside index is never touched. Output uses the wp-add store (matches the coarse path; byte-identical to both in the default double build, and the more accurate choice in the untested --mixed config). GPU (V100): all 12 subcycle-single-level + multi-level goldens (1D/2D, static/dynamic/ tiled) byte-identical. (A GPU-only present-table trap from array-element loop bounds was caught by this validation and fixed by hoisting the bounds to scalars.)
The coarse-side creg capture (islot loop, region/sidx frame, per-face ownership gating) and the child creg capture (kc loop, parent-fine frame, both faces) in s_amr_capture_boundary_flux were near-duplicate: same advective + total-flux-viscous + chemistry-species/energy math over two frames. A future physics term added to one side but not the other would silently break conservation on one path (the review's concern). Extract two shared kernels: s_amr_capture_creg_dense (dense eq range, per-face gated, accumulate-or-overwrite via merge(creg,0,acc)+cf*flux) covers advective and viscous; s_amr_capture_creg_chem covers the species-seq + conditional-energy chemistry capture. The coarse and child branches now just compute their frame and call the same kernels (net ~-190 lines). The merge idiom is byte-identical to the old if(accum)/else and never reads uninitialized creg (merge selects, no arithmetic). GPU (V100): 20 goldens across inviscid / viscous / chemistry / bubbles / QBMM / multi-level / subcycle / 3D all byte-identical.
…ware extents, nesting guard Review of PR #6 (amr-multilevel). Correctness/safety fixes: - static builder (s_amr_build_static_multilevel): fail-closed guards for an inverted L2 box (small parent) and for an L2 whose L0-extent > amr_maxc_fit/2 (would overrun the creg register 0:amr_maxc_fit-1 in the L2->L1 reflux capture) — the dynamic regrid path already caps/clamps. - checker: gate multi-level+IB at ALL rank counts (was np>1 only); gate static AMR (amr_regrid_int=0) to amr_max_level<=2. - proper-nesting guard in the regrid: abort fail-closed if a level>=2 block overlaps != 1 parent-level block (gather/reflux take the FIRST overlap; a tile straddling a parent-tile seam would silently couple to one parent — wrong BC + leak). - stretched-grid reflux-to-parent: per-face parent-fine dx mirroring s_amr_apply_reflux_state (byte-identical on uniform grids). - level-aware extents: tower load-weight and the regrid stash old_ext use 2**level (level-2 is 4x its L0 footprint); level-1 byte-identical. Fast-path also compares box_level; guard tower roll-up against amr_block_level(0). - fix stale comment: the child creg captures total flux (advective+viscous+chem), not advective only, so viscous/chem multi-level conserves. Validated: 8/8 multi-level AMR goldens byte-identical on GPU (V100); guards are fail-closed.
The amr_max_level description in case.md, descriptions.py, and amr_multilevel.md still said only level 1 was supported / multi-level was 'planned', contradicting the shipped, golden-tested amr_max_level=2 (static up to 2, dynamic regrid deeper). Correct all three.
…path (MFlowCode#691) s_amr_copy_parent_patch synced the gathered coarse patch amr_cg device->host after every parent gather, but the runtime C/F ghost-fill (s_amr_fine_stage_advance lock-step, s_amr_fill_fine_ghosts subcycle) reads amr_cg on the DEVICE (filled by the copy kernel); only the init/regrid host prolong and the restrict-prolong self-test consume it on the host. Thread a to_host flag through s_amr_gather_from_parent_field -> s_amr_copy_parent_patch (= .not. pull_host: init/regrid true, runtime false; subcycle call sites explicit false) so the D2H runs only when a host consumer follows. Removes a per-substep, per-level>=2-block whole-patch device->host transfer. Validated: 8/8 multi-level AMR goldens byte-identical on GPU (V100).
…t round-trip) s_amr_gather_coarse_patch (the L0->L1 coarse-BC gather) pulled q_coarse device->host, unpacked it into amr_cg on the host, then pushed amr_cg host->device - a full round-trip every RK stage per fine block. At num_procs==1 the sole owner holds every covered coarse cell, so copy q_coarse (device) -> amr_cg (device) with a DEVICE kernel over the in-domain patch (same index map as s_amr_unpack_patch), and skip the q_coarse pull. Gated on pull_host (runtime): at init/regrid q_coarse's device copy may be stale, so those keep the host path. np>1 keeps the host pack for the MPI P2P gather. amr_cpat_off hoisted to scalars for present-table safety (mirrors s_amr_copy_parent_patch). With this + the MFlowCode#691 amr_cg fix, the np=1 AMR advance's coarse-BC gather (L0->L1 and L1->L2) stays entirely on-device per stage - no host round-trips except the seam halo. Validated: full 56-test AMR golden suite byte-identical on GPU (V100).
…k/active copies were non-conforming at amr_max_level>=2)
…evel nesting (IB path left it at the cluster count, overrunning on child append)
The granularity-floor section said a shallow hierarchy leaves ranks 'idle by construction', and I read the step-5 np=8 measurement the same way, reporting 'idle_ranks 6 of 8' as 75 percent of the machine idle. That is wrong. The counter is per-LEVEL: a rank holding no level-L block still owns level-0 work (level 0 covers every rank) and may hold blocks at other levels; even the TOTAL line means only 'owns no fine block'. So the number does not explain the 18 percent parallel efficiency - that still needs measured per-rank time, which is the point the metric section already makes. Also mark step 5 landed while stating that its m_rank_timing half, the half that measures idleness rather than inferring it, is not done.
Removing the multi-level subcycle PROHIBIT left twelve lines of explanation sitting in a list of checks with nothing after them - a comment justifying why something is NOT checked, in the file whose job is checking. The content is already in the two places it belongs: s_amr_advance_children's docstring (the level-wide walk and the interposed halo) and the C45DBB52 test comment (the amr_cpat_mar margin argument).
be94db3 shipped the per-level cap saying it was verified by reading the slot arithmetic, not by running, because no golden covers amr_max_level >= 3. It is now verified by running, and the evidence belongs where the next reader will be standing. m = 255, amr_max_level = 3, amr_buf = 48, np = 1: the fixed /2 keeps ONE oversized level-3 box where the per-level cap keeps 2, and the run aborts in s_amr_free_slot with a Fortran 'Invalid descriptor' and a core dump; the per-level cap exits 0. Two earlier counterfactuals produced byte-identical output and proved nothing - m = 63 was too small, and scaling the grid without scaling amr_buf left the boxes tracking the feature rather than the domain, far below either cap. A635AA56's comment already says it is the BUFFER that widens the box; applying that is what made the cap bind.
…nchmarking The balancer only becomes observable when a level holds many boxes, and stripe cases collapse to ~2 boxes per level by np=8 - at which point every owner mapping produces an identical assignment. Scattered blobs give many disjoint tag clusters, hence many boxes per level. Hardcoded rather than an analytic patch on purpose: analytic expressions compile into case.fpp, so every variation costs a rebuild, whereas a(2:5) (count, seed, radius, amplitude) are read at run time and sweep freely against one binary. Centres come from an additive-irrational (Weyl) sequence, NOT random_number: each rank fills only its own cells, so the IC must be a pure function of position or ranks disagree at the subdomain seam and the decomposition stops being exact. Purely additive - a new case in a select, unreachable unless a case sets hcid = 299 - so no existing golden can enter it. Verified: all four targets build (the file compiles into pre_process via m_icpp_patches and simulation via m_ib_patches), pre_process runs it clean at 511x255 on 2 ranks. Also revises the plan doc: step 5 landed, acceptance criterion 2 promoted to a runnable step, and a retraction of this file's own earlier claim that m_rank_timing was unimplemented - it is fully wired at eight call sites.
…uption) s_amr_regrid's 'brand-new region' branch - a level>=2 child whose region has no old fine data to cluster - emitted its box straight into boxes() without ever consulting amr_maxc_fit. It is the ONLY box-emitting path that skips s_amr_tile_box. The inset bounds the child as a FRACTION of its parent (span/4 each side), but the slot coord arrays are allocated ONCE to amr_ref_ratio*amr_maxc_fit, so the child must be bounded in ABSOLUTE cells. s_amr_build_block_coords then sizes fcb from the block's TRUE extent and writes past x_cb. Under -O2 that silently scribbles past a heap allocation every regrid and surfaces much later as 'corrupted size vs. prev_size' inside an unrelated free() - the crash site is nowhere near the bug site, which is why three passes of reading the seam/overlap/tiling code cleared every candidate. Integer division is the trigger: s_amr_tile_box splits a wide region into 63 and 64, not 64 and 64, and a parent of span 63 gives ins = 63/4 = 15 and a child of span 33 against a level-2 cap of 32, while span 64 gives exactly 32 and is fine. A one-cell difference in the parent flips it. Introduced by me in cfdd284 (step 4): s_amr_build_block_coords replaced a parent-relative computation with a global-extent one so a rank could build coords for a block whose parent it does not own, and the destination sizing did not follow. VERIFIED: bounds-checked CPU build (gfortran --no-gpu --debug, the ONLY configuration with -fcheck - cmake/GPU.cmake gives LLVMFlang none) traps at m_amr.fpp:584 'Index 128 of dimension 1 of array fcb above upper bound of 127' without this change, and with it the span-33 box becomes 17+16 and the run exits 0.
amr_max_grid_size had ZERO test coverage - not one golden or example case set it, only the schema, validator and definitions - despite being the mechanism the rank-independent-cap work rests on. Any defect reachable only through the pinned-cap path was invisible to CI, and one was: the brand-new-region box emitter skipped s_amr_tile_box and corrupted the heap (cfbaceb). VERIFIED to fail without that fix: 8 over-cap boxes and, under a bounds-checked build, 'Index 32 of dimension 1 of array fcb above upper bound of 31' (the trap index tracks the cap, max_f1 = 2*amr_max_grid_size - 1). Both knobs are load-bearing and narrow. amr_max_grid_size = 16 makes s_amr_tile_box split a wide region into 16 and 15, and span 15 gives ins = 15/4 = 3 and a child of 9 against a level-2 cap of 8; INTEGER DIVISION is the trigger, so a cap of 20 splits evenly (20/20 -> child 10, cap 10) and does not reproduce. The blob IC is equally load-bearing - a Sod-like patch grows level-2 regions that already have fine data, so the brand-new-region branch never fires and the bug is unreachable however the cap is set; three separate small 1D Sod candidates passed for exactly that reason. Also records the caveat in the plan doc: step 2's measured speedup was collected on this path, so it is no longer a clean measurement. 72/72 AMR goldens, precheck 7/7.
Two additions, both from today's heap-corruption hunt (cfbaceb). (1) s_amr_check_box_caps aborts if any level>=1 box exceeds amr_maxc_fit/amr_ref_ratio**(lev-1), checked once where the box set is final rather than trusting each emitter to have routed through s_amr_tile_box. A violation is otherwise silent and catastrophic - the coord arrays are sized to the cap, so an over-cap box writes past x_cb and corrupts the heap every regrid, surfacing much later inside an unrelated free(). VERIFIED BOTH WAYS: on the unfixed reproducer it prints '[amr] box cap violated: level 2 dim 1 span 9 > cap 8' and aborts by name instead of corrupting memory; and 71/71 existing AMR cases pass with it compiled in, zero false aborts. The false-positive hazard was real - in 1D/2D amr_maxc_fit(2:3) is 1, so the level-2 cap divides to 0 and a naive check would abort every 1D golden; it mirrors s_amr_tile_box's own max(tc,1) floor. (2) [amr-balance] now prints boxes_max/mean beside the weight imbalance, because measured per-rank time tracks BOX COUNT rather than cells: at np=8 the cell weight sat at 1.050 while box count was 1.308 and rank_time 1.259. Printing both makes that falsifiable in one run. 72/72 AMR goldens, precheck 7/7.
2 tasks
s_amr_sfc_cut advances a rank when cum >= tgt, where cum is an n-term ACCUMULATION and tgt = (r+1)*total/num_procs is CLOSED FORM over another n-term sum. At an exact share boundary the two differ by rounding rather than by intent, so the comparison turns on 1 ULP. This has been correct only by luck: every cost term to date is integer-valued (footprint cells, K_ib = 2, K_pc = 3 x integer iteration counts), so 4w and 32w/8 agree bit for bit. ANY fractional cost term breaks it, silently - ranks come out unbalanced with no error and no diagnostic. MEASURED with a fractional term in cost(k): 32 IDENTICAL level-1 weights (9.038222222E+03, verified by printing them) over 8 ranks split 5/3 instead of 4/4, reporting max/mean 1.250 where the same case with integer weights reports exactly 1.000. Adding a tolerance of O(n) ULP of the target - the accumulated rounding bound - restores 1.000 while leaving the genuine imbalance at the other level untouched (1.106, unchanged to three decimals). Far from a share boundary the tolerance is ~1e-12 relative and the greedy is unchanged, which is what 72/72 AMR goldens confirm: no existing case moves. Found while adding a per-box cost term; that term is NOT in this commit - the fragility predates it and is worth fixing on its own.
Fine blocks occupy slots (l0_slot_off, amr_num_blocks]; slots [1, l0_slot_off] are the L0 TILE prefix. At init s_amr_assign_block_owners runs BEFORE s_l0_tiles_init, so those prefix slots are still uninitialized - amr_block_level reads 1 and amr_region_lo is all zeros, i.e. Morton key 0. The assigner looped from slot 1 and therefore fed phantom key-0 'level-1 blocks' into the level-1 cut and split them across ranks. A key-0 block can only ever resolve to rank 0 (the cut is non-decreasing and the search returns the first r with key <= cut(r)), so any phantom placed on a higher rank is unrecoverable and s_amr_validate_owner aborts with 'SFC cut-point owner disagrees with amr_block_owner'. Observed directly by dumping the block table in a coexist np=2 run: fine_cut(:,1) = [0, 33792], owner_cut = [-1, -1] (tile cut not yet built), blocks 1-8 all lev 1 / key 0 / lo (0,0,0), blocks 9-10 the real fine blocks at keys 5120 and 33792. Blocks 7 and 8 were stored as rank 1 while f_amr_owner returned 0. LATENT, NOT NEW: with the existing weights every phantom happened to land on rank 0 and the validator agreed by luck. It surfaced only when an experimental per-box cost term moved the cut - the same shape as the ULP tie in 2051aa1, a fragility surviving on arithmetic that happened to be benign. Fix: the assigner and the validator both restrict to (l0_slot_off, amr_num_blocks], the validator skipping the prefix only while amr_owner_cut is unbuilt (the tile-init call site populates both cuts and validates there). Also records in the plan doc that s_amr_validate_owner - marked TRANSITIONAL in-source - is load-bearing and caught this immediately. 72/72 AMR goldens, precheck 7/7.
K_box entered m_constants.fpp as part of an experimental fixed per-box cost term. The consuming code in m_amr.fpp was reverted, but the constant lived in a different file and a git add -A swept it into c3364a5, leaving a parameter with no consumers and a comment citing a calibration sweep that nothing acts on. The experiment itself did not survive scrutiny: at a properly loaded size (75.5M cells, 512 level-1 boxes over 8 ranks = 64 boxes/rank) K_box=8 and K_box=0 measure 1.035 and 1.052 rank-time imbalance, a difference inside the run-to-run variance. The earlier effects were measured on cases small enough that the granularity floor made the distribution lumpy, and on an IC whose blob centres come from a low-discrepancy Weyl sequence - evenly spread by construction, so the workload is inherently balanced and cannot discriminate between cost models. What the experiment DID produce is two latent partitioner bugs it exposed by perturbing the weights, both fixed independently: the ULP-fragile SFC boundary (2051aa1) and the phantom tile-prefix slots (c3364a5).
… questions Three claims in this file were false after the day's measurements. (1) Step 5's conclusion 'box supply binds, not the owner mapping' was inferred from box counts and model imbalance on a starved case with no measured counterpart - retracted. (2) Step 7 was listed as 'measured good, blocked by a coexist regression'; the regression is fixed (c3364a5) and K_box was rejected for an entirely different reason - it delivers ~2% at production scale, inside this machine's run-to-run variance. (3) The K_box table quoted starved-GPU numbers as though they were the result. Replaced with the regime measurements: which metric predicts runtime depends on how much work a box carries, and at 75.5M the two converge (weight 1.058, box count 1.121) so the objectives barely conflict. Includes a prediction of mine that FAILED - I expected K_box to make balance worse at scale by steering toward box count, and it improved it slightly - kept because it constrains the model. Requeues the open questions around what is actually unknown: whether the 45% np=8 ceiling is MFC's or this branch's (portable uniform case on master vs branch, gen_uniform.py), what AMR costs relative to uniform at a loaded size (the AMR np=1 point exceeded the harness cap, so there is no baseline), and a note that the per-box-overhead argument for the batching arc rests on efficiency figures later found contaminated by starved GPUs and run_time_info serialisation.
… accordingly Paired experiment on the identical portable uniform case (gen_uniform.py, standard geometry patches, no hcid, no AMR in the path), MFC master bfdc8f5 vs this branch, 3 reps alternating between trees so drift hits both arms equally: branch/master median ratio 1.022 at np=4 and 1.023 at np=8, against within-tree run-to-run spreads of 6.9% and 11.1%. The branch costs ~2% on a uniform problem, indistinguishable from noise, and uniform efficiency is ~60% at np=8 on BOTH trees. An earlier single-run comparison suggested 15% at np=4; three reps put it at 2%, so that was an artifact. Two consequences. First, no AMR-side work can move the np=8 ceiling - it belongs to the uniform solver on this machine. Second, the batching arc's premise (per-block overhead dominates) was argued from efficiency figures since found contaminated by starved GPUs and by run_time_info forcing a device sync every step; against a 60% ceiling the recoverable headroom is smaller than assumed, and it should not be treated as established. The queue is reordered around what is actually unknown: what AMR costs relative to uniform at a loaded size (the np=1 AMR point blew the harness cap, so there is no baseline), and multi-node scale invariance, which is the real exascale question and which single-node np<=8 structurally cannot answer. Also records the measurement standard this session earned: at 11% within-tree spread, any A/B claiming less than ~10% needs reps.
…with the GLOBAL box set
This was known and never written down as a work item. It appeared only as one clause inside 'The problem, measured' ('every per-box collective in the regrid rebuild loop runs over all boxes on all ranks') with no design entry, no queue entry and no owner, so it was invisible when planning. Audited against the code and quantified. Three costs grow with the global box count while the whole strategy demands boxes_per_level >> num_procs: (1) ONE GLOBAL COLLECTIVE PER BOX PER REGRID - s_set_amr_fine_geometry ends in s_mpi_allreduce_integer_max and is called from do k = 1, nboxes in s_amr_regrid_rebuild_slots; 512 collectives today, 10^5 at the target, and amr_block_batching already measured it at 7.4-13 ms per call because it absorbs the spread in the owner-only work before it, so the cost grows with rank count too. (2) the SFC cut's insertion sort is O(n^2) with a comment saying 'n small' - 5e9 ops at 10^5 boxes. (3) six metadata arrays sized amr_max_blocks are replicated on every rank. Limit 1 is trivially removable: every per-box answer is immediately OR-ed into any_xchg and only the accumulator survives (m_amr_regrid.fpp:1456), so the reduction hoists out of the loop for ~10 lines. Also records which half of the AMReX model is adopted - ownership, cap, per-box scratch, per-level mapping, redundant communication-free mapping and P2P gather are ALL in place; the per-box reduction is the single gap, and it was never a design decision. Queue reordered with the hoist as item 8. None of this is reachable on a single node: 512 boxes is three orders of magnitude below where it binds, so measurement could not have found it and a code audit did.
…rement sections The file had grown to 612 lines and 27 sections by accretion, with the primary open work item buried in the middle and THREE overlapping cost-model sections, two of which carried numbers already retracted elsewhere in the same document. Reorganised to 511 lines and 12 sections. What actually blocks exascale (the per-box collective, the O(n^2) cut, the replicated metadata) now comes immediately after Status instead of sitting between two design sections. Three sections were replaced by a single note recording why they are not evidence: 'Balance is no longer the limiter' quoted 62%/38% efficiency figures produced with run_time_info on, which forces a device sync every step and measured 6% on a uniform control; 'The cost model weighs the wrong quantity' reported measured time tracking box count, which is real but confined to the starved regime; and the step-4 A/B ran at a size where the uniform control itself reached only 18% efficiency. Kept rather than deleted so a reader meeting them in git history knows not to trust them. Added a reading guide and fixed two cross-references left dangling by the removals.
s_set_amr_fine_geometry ended in a global s_mpi_allreduce_integer_max and is called from do k = 1, nboxes in the regrid rebuild loop, so a regrid cost nboxes global collectives. Every per-box answer was immediately OR-ed into a single accumulator and only the accumulator survived (m_amr_regrid.fpp:1456), so the reduction had no reason to be inside the loop. It now ORs into a module accumulator amr_xchg_bad and s_amr_reduce_xchg_flag performs ONE allreduce to close the scan; all five call sites close their scan explicitly. This also FIXES A LATENT BUG at two of them: the L0-tile loop in s_initialize_amr_module and both restart loops kept only the LAST block's answer rather than the OR, so a block needing the coarse-ghost exchange could be masked by a later one that did not, silently skipping an exchange the fine advance depends on. The accumulator makes that impossible by construction. EXPECT NO SPEEDUP AT CURRENT SIZES, and that is the point: amr_block_batching measured that batching these allreduces buys nothing at 14-21 boxes because the 7.4-13 ms per call is absorbing load-imbalance spread, not reduction cost. That measurement is regime-limited - with nboxes collectives the cost has a floor of nboxes x latency independent of any imbalance, ~0.5 s per regrid at 1e5 boxes, and the design targets boxes_per_level >> num_procs. Both docs now state the reconciliation rather than contradicting each other. Also fixes the Documentation CI failure: amr_per_level_distribution.md was the only AMR doc without a @page declaration, so an earlier workaround that referenced it as a backticked filename produced a dead auto-generated link from amr_block_batching.html. 72/72 AMR goldens, precheck 7/7.
…d by device copies
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.