Resume and extend Bayesian (bumps-DREAM) refinements

docs/dev/adrs/accepted/analysis-cif-fit-state.md

@@ -29,7 +29,7 @@ Analysis-owned fit state needs to persist:
 - deterministic correlation summaries
 - minimizer-specific fit outputs on the paired `_fit_result.*` category
 - per-parameter posterior summaries on `_fit_parameter`
-- large posterior arrays and plot caches in `analysis/results.h5`
+- large posterior arrays and plot caches in `analysis/mcmc.h5`
 
 Committed model parameter values and uncertainties already persist in
 structure and experiment CIF files through the accepted free-flag CIF
@@ -44,7 +44,7 @@ projection. This ADR defines that narrower saved projection.
 
 Persist analysis-owned fit state as explicit analysis categories in
 `analysis/analysis.cif`, with large posterior arrays stored in
-`analysis/results.h5`.
+`analysis/mcmc.h5`.
 
 Do not add a dedicated `_fit_state` category or
 `_fit_state.schema_version`. Persisted fit state is detected from
@@ -235,12 +235,12 @@ metadata from `_minimizer.*`.
 
 ### Posterior sidecar
 
-Persist large posterior arrays in `analysis/results.h5` using `h5py`.
-This includes canonical posterior arrays and saved distribution, pair,
-and predictive cache arrays. The HDF5 file is self-describing; no CIF
+Persist large posterior arrays in `analysis/mcmc.h5` using `h5py`. This
+includes canonical posterior arrays and saved distribution, pair, and
+predictive cache arrays. The HDF5 file is self-describing; no CIF
 manifest rows or sidecar filename tags are persisted.
 
-The sidecar filename is fixed to `results.h5` inside the project
+The sidecar filename is fixed to `mcmc.h5` inside the project
 `analysis/` directory.
 
 If the sidecar is missing on load, summary rows in

docs/dev/adrs/accepted/bayesian-resume-and-mcmc-sidecar.md

@@ -0,0 +1,201 @@
+# ADR: Bayesian Resume (DREAM) and MCMC Sidecar Naming
+
+## Status
+
+Accepted.
+
+## Date
+
+2026-06-15
+
+## Group
+
+Analysis and fitting.
+
+## Context
+
+EasyDiffraction supports two Bayesian MCMC minimizers: **emcee** and
+**bumps DREAM**. Resume (continue/extend a previously sampled chain
+across sessions) is implemented for **emcee only**:
+
+- `MinimizerFitOptions.resume` / `extra_steps` and the matching
+  `FitterFitOptions` already exist and are engine-agnostic.
+- `MinimizerBase.fit()` raises
+  `NotImplementedError("…does not support resume")`; `EmceeMinimizer`
+  overrides `fit()` to implement it.
+- emcee persists its raw chain **live during sampling** via
+  `emcee.backends.HDFBackend(name='emcee_chain')` into the project's
+  `analysis/results.h5` sidecar. Resume reads the last state from that
+  HDF5 group and runs `extra_steps` more iterations.
+- `BumpsDreamMinimizer` runs `FitDriver.fit()`, captures
+  `driver.fitter.state` (a bumps `MCMCDraw`), but **discards** it. Only
+  the _derived_ posterior arrays reach the sidecar via
+  `write_analysis_results_sidecar()`. The raw sampler state is never
+  persisted, so there is nothing to resume from.
+
+bumps DREAM **does** support resume — `FitDriver.fit(fit_state=…)` plus
+`bumps.dream.state.save_state`/`load_state` (gzipped `.mc` text files)
+or `DreamFit.h5dump`/`h5load` (HDF5). The capability is unused because
+the caller must persist the state explicitly; emcee only looks
+"automatic" because its backend streams to disk during the run.
+
+`easyscience/core` PR #257 ("Bayesian extend/resume") is a reference
+implementation for the DREAM-side mechanics: it surfaces the `MCMCDraw`
+state in the result, accepts a `resume_state`, recovers the population
+scale factor from `state.Npop`, validates parameter count/order, deep-
+copies the state before fitting (bumps mutates it in place), and
+documents the **ring-buffer contract** — DREAM keeps only the last
+`samples` draws, so extending an M-draw chain by N means
+`samples = M + N, burn = 0`.
+
+Separately, the sidecar filename `results.h5` is misleading: the file
+only ever holds MCMC/Bayesian content (posterior arrays, distribution
+and pair caches, posterior-predictive sets, and emcee's raw chain) and
+is created only for Bayesian minimizers — deterministic least-squares
+results live in CIF, not here.
+
+Two accepted ADRs currently fix the sidecar name and the one-file rule:
+
+- [`analysis-cif-fit-state.md`](../accepted/analysis-cif-fit-state.md) —
+  "The sidecar filename is fixed to `results.h5`".
+- [`minimizer-category-consolidation.md`](../accepted/minimizer-category-consolidation.md)
+  — "There is exactly **one** sidecar file per fit, regardless of
+  minimizer: `analysis/results.h5`".
+
+## Decision
+
+### 1. Extend resume to bumps DREAM, consistent with emcee
+
+`BumpsDreamMinimizer` gains resume parity with `EmceeMinimizer` behind
+the existing engine-agnostic API:
+`analysis.fit(resume=True, extra_steps=N)`. The owner-level surface and
+`MinimizerFitOptions` do not change. Internally:
+
+- `BumpsDreamMinimizer` overrides `fit()` (like emcee) instead of
+  inheriting the `NotImplementedError` guard.
+- On a fresh run the resumable `MCMCDraw` state is **captured** and
+  written to the sidecar at `project.save()`.
+- On resume the state is loaded, deep-copied (bumps mutates it in
+  place), validated against the current model, and passed as
+  `FitDriver.fit(fit_state=…)`.
+- The unified `extra_steps=N` is translated to DREAM's ring-buffer
+  semantics: request `samples = current_draws + N` with `burn = 0`,
+  recovering the population scale factor from `state.Npop` so the
+  population is unchanged. emcee keeps its existing append semantics;
+  the user-facing contract (`resume=True, extra_steps=N` adds N) is the
+  same for both engines.
+
+Resume validation mirrors emcee's `_validate_resume` but for DREAM
+quirks: matching fitted-parameter count and population, and — because we
+control persistence — **name-based** parameter validation (we store our
+parameter names alongside the state, avoiding core's positional-only
+fallback).
+
+The DREAM minimizer also gains a user-facing **`chains` alias** for the
+existing `population_size` setting (an approved API addition): `chains`
+is the discoverable name for the population _scale factor_ — bumps
+creates `ceil(chains · n_parameters)` parallel chains. `chains` and
+`population_size` are two names for **one** descriptor (shared storage),
+so they are always value-consistent and cannot disagree; no separate
+`population` field is added (avoiding a third name for the same value).
+
+### 2. Persist resumable raw sampler state per engine, in one sidecar
+
+Both engines write their raw, resumable state into a **single** sidecar
+file, distinguished by an **engine-keyed HDF5 group**:
+
+- emcee: the existing `emcee_chain` group (unchanged mechanics).
+- DREAM: a new top-level **`dream_state`** group containing the
+  `MCMCDraw` written by `DreamFit.h5dump(group, state)`, plus a
+  **`param_names`** dataset (the fitted-parameter names, in order) for
+  name-based resume validation. This layout is fixed by this ADR, not
+  deferred to implementation.
+
+The derived posterior arrays (`/posterior/*`, caches, predictive sets)
+continue to be written by `write_analysis_results_sidecar()` as today.
+
+**State lifecycle (one sidecar, several engines).** A _fresh_
+(non-resume) fit clears **all** raw sampler-state groups (both
+`emcee_chain` and `dream_state`) before writing — consistent with the
+existing rule that `analysis.fit()` truncates the sidecar (see
+`minimizer-category-consolidation.md` §4). Clearing _every_ group, not
+just the active engine's, is what prevents the stale-state trap: an
+emcee fit, then a fresh DREAM fit, then `emcee resume=True` must **not**
+resume the original emcee chain. Resume detection and resume then read
+**only** the group matching the active minimizer
+(`analysis.minimizer.type`). On explicit `resume=True` a missing or
+malformed active-engine group is a clear error; otherwise it is ignored
+and the fit starts fresh. `undo_fit` clears the raw-state group(s) the
+same way it already clears the sidecar.
+
+**Relocating a saved project (`save_as`).** `project.save()` rebuilds
+the derived sidecar arrays from memory but cannot reconstruct the raw
+sampler state, which only ever exists on disk. Relocating a project with
+`save_as` therefore copies the raw-state groups (`emcee_chain`,
+`dream_state`) from the source sidecar into the destination before the
+derived arrays are rewritten. Without this, a resume after `load` +
+`save_as` — the flow both Bayesian resume tutorials use — would find no
+chain to extend. This makes resume genuinely survive a load/relocate
+round-trip for both engines, as required above.
+
+### 3. Rename the sidecar `results.h5` → `mcmc.h5`
+
+The sidecar is renamed to reflect its content. It remains **one file per
+fit** (the consolidation ADR's invariant holds; only the name changes).
+The filename is defined once as a single constant and the two duplicated
+string literals in `analysis/fitting.py` and `analysis/analysis.py` are
+replaced by that constant / a shared path helper.
+
+### 4. Amendments to accepted ADRs
+
+The rename touches **every** repository reference to `results.h5`, not
+only the two ADRs that fix the name. On implementation, a
+`git grep -n 'results\.h5'` sweep updates all source, docs, tests, and
+tracked fixtures (excluding generated/transient outputs). Concretely
+this currently spans:
+
+- Accepted ADRs: `analysis-cif-fit-state.md` (filename `results.h5` →
+  `mcmc.h5`), `minimizer-category-consolidation.md` (filename + the
+  per-engine-state-groups clarification above), `undo-fit.md`,
+  `minimizer-input-output-split.md`, `runtime-fit-results.md`,
+  `edstar-project-persistence.md`, and the `docs/dev/adrs/index.md`
+  rows.
+- Suggestion ADR `fit-output-files-and-data-exports.md`.
+- User docs: `docs/docs/cli/index.md`,
+  `docs/docs/user-guide/{concept,data-format}.md`,
+  `docs/docs/user-guide/analysis-workflow/{analysis,project}.md`.
+- Source: `io/results_sidecar.py`, `analysis/fitting.py`,
+  `analysis/analysis.py`, `__main__.py`.
+- Tests referencing the literal, and any tracked project fixtures.
+
+## Consequences
+
+- Resume/extend works identically for emcee and DREAM from the user's
+  point of view; the long-running Bayesian tutorials gain a DREAM resume
+  page mirroring the emcee one.
+- The sidecar name communicates intent (`mcmc.h5`), and the filename
+  lives in exactly one place.
+- The project is in beta (no legacy shims): committed project fixtures,
+  tutorials, and any test referencing `results.h5` are regenerated /
+  updated to `mcmc.h5`; old saved projects must be re-saved.
+- No new third-party dependency: `bumps` is already a dependency and its
+  `MCMCDraw`/`h5dump`/`h5load` API is used directly.
+- DREAM resume persists a second representation of the chain (raw state)
+  alongside the derived posterior; the sidecar grows modestly.
+
+## Alternatives Considered
+
+- **Per-engine filenames (`mcmc_emcee.h5`, `mcmc_bumps-dream.h5`).**
+  Rejected: forces the load/undo/clear paths to resolve "which file"
+  from the active minimizer and breaks the one-sidecar invariant for
+  little gain over engine-keyed groups in one file.
+- **Persist DREAM state as bumps `.mc.gz` text triples (as
+  `easyscience/core` does).** Rejected: three extra files per fit and a
+  dependence on bumps' text parser, which has a known 1.0.4 regression
+  (`load_state` collapses single-row buffers to 1-D). `DreamFit.h5dump`
+  into the existing sidecar avoids both.
+- **Keep the `results.h5` name.** Rejected: misleading; the file is
+  MCMC-only.
+- **In-memory-only resume (no disk persistence).** Rejected: resume must
+  survive `project.save()`/load across sessions, which is the whole
+  point.

docs/dev/adrs/accepted/edstar-project-persistence.md

@@ -97,7 +97,7 @@ project_dir/
 |-- analysis/
 |   |-- analysis.edi
 |   |-- results.csv
-|   `-- results.h5
+|   `-- mcmc.h5
 `-- reports/
     `-- <project>.cif
 ```
@@ -108,7 +108,7 @@ continue to use `_easydiffraction_*` extension categories inside report
 CIF.
 
 Edi governs the `*.edi` files only. Existing non-STAR analysis artifacts
-keep their current formats: `analysis/results.h5` remains the binary
+keep their current formats: `analysis/mcmc.h5` remains the binary
 fit-result sidecar, and `analysis/results.csv` remains the tabular
 sequential-fit output used by plotting and user inspection.
 

docs/dev/adrs/accepted/minimizer-category-consolidation.md

@@ -137,46 +137,56 @@ populated by deterministic or Bayesian fits as appropriate. Per-
 parameter posterior order is the order of the `_fit_parameter` rows
 themselves; no separate parallel loop is needed.
 
-### 4. Heavy posterior arrays live in `analysis/results.h5`, not in CIF
+### 4. Heavy posterior arrays live in `analysis/mcmc.h5`, not in CIF
 
 Posterior chains, KDE / distribution caches, pair-plot caches, and
 predictive datasets are large arrays unsuited to CIF. The existing
-`analysis/results.h5` sidecar absorbs all of them. The corresponding
+`analysis/mcmc.h5` sidecar absorbs all of them. The corresponding
 manifest categories (`_bayesian_distribution_cache`,
 `_bayesian_pair_cache`, `_bayesian_predictive_dataset`) are removed from
 CIF entirely — the HDF5 file is self-describing.
 
 There is exactly **one** sidecar file per fit, regardless of minimizer:
-`analysis/results.h5`. No CIF tag stores the sidecar path. The file uses
+`analysis/mcmc.h5`. No CIF tag stores the sidecar path. The file uses
 namespaced top-level groups:
 
 ```
-analysis/results.h5
+analysis/mcmc.h5
 ├── /posterior/            # canonical posterior chains, log-prob (all Bayesian samplers)
 ├── /distribution_cache/   # KDE / 1-D distribution plots
 ├── /pair_cache/           # pair-plot grids
 ├── /predictive/           # posterior-predictive datasets
-└── /emcee_chain/          # emcee HDFBackend live state (emcee runs only)
+├── /emcee_chain/          # emcee HDFBackend live state (emcee runs only)
+└── /dream_state/          # bumps-DREAM MCMCDraw + param_names (dream runs only)
 ```
 
+Each engine persists its **resumable raw sampler state** in its own
+top-level group (`/emcee_chain/`, `/dream_state/`); see
+[`bayesian-resume-and-mcmc-sidecar.md`](bayesian-resume-and-mcmc-sidecar.md).
+
 **Lifecycle rule: a new fit overwrites the file.** Mixing partial
 results from different minimizers — or from the same minimizer with
 different settings or a different free-parameter set — is the most
 common source of "stale plot" confusion. To prevent this, calling
-`analysis.fit()` truncates `analysis/results.h5` (recreating it with the
+`analysis.fit()` truncates `analysis/mcmc.h5` (recreating it with the
 new run's groups). The user is shown a `log.warn(...)` message the first
 time a fit is started while a populated sidecar exists, naming the file
 and stating that previous results will be overwritten.
 
 Resume is the only exception: `analysis.fit(resume=True, extra_steps=N)`
-opens the existing file in append mode and extends the chain. Resume is
-rejected with a clear error if the active minimizer does not support it,
-if `results.h5` is missing, or if the stored chain's parameter set does
-not match the current one.
+opens the existing file in append mode and extends the chain. It is
+supported by both emcee and bumps-DREAM
+([`bayesian-resume-and-mcmc-sidecar.md`](bayesian-resume-and-mcmc-sidecar.md)),
+each reading only its own state group. Resume is rejected with a clear
+error if the active minimizer does not support it, if `mcmc.h5` is
+missing, or if the stored chain's parameter set does not match the
+current one. Because a fresh (non-resume) fit truncates the file, no
+stale raw-state group from a previous engine can be resumed by accident.
 
 For deterministic runs the Bayesian groups are absent and the sidecar
-file may not exist at all. For non-emcee Bayesian runs the
-`/emcee_chain` group is absent.
+file may not exist at all. Only the active engine's raw-state group is
+present after a run (`/emcee_chain` for emcee, `/dream_state` for
+DREAM).
 
 ### 5. Unified, verbose attribute names with internal mapping
 
@@ -377,8 +387,7 @@ _fit_result.best_log_posterior        -1237.89
 ```
 
 emcee's resumable chain state lives in the `/emcee_chain` group of the
-same `analysis/results.h5` file (see §4). No sidecar path appears in
-CIF.
+same `analysis/mcmc.h5` file (see §4). No sidecar path appears in CIF.
 
 ## Superseded Selector Layout
 

docs/dev/adrs/accepted/minimizer-input-output-split.md

@@ -419,7 +419,7 @@ under `_minimizer.*`, the duplications with `fit_result` remain, and the
 ### C. Move outputs into the runtime `fit_results` object, not a CIF category
 
 Persist only settings in CIF; outputs live in `analysis.fit_results` at
-runtime and `analysis/results.h5` on disk. Rejected because the small
+runtime and `analysis/mcmc.h5` on disk. Rejected because the small
 scalar outputs (success, χ², runtime, R̂) are exactly what users want to
 read from CIF without unpacking HDF5, and the consolidation ADR
 explicitly puts them in CIF (`_minimizer.*` today).