feat(fingerprint): switch to new fingerprint algorithm [3/3]

.github/copilot-instructions.md

@@ -31,6 +31,8 @@ Documentation structure: `docs/user/reference/cli/` (auto-generated CLI docs, re
 
 The TOML config files in `defaultconfigs/` are loaded via `internal/projectconfig/`.
 
+**CRITICAL when editing config structs or fingerprinting:** Adding a field to `ComponentConfig` or its component structs (or touching `projectV1` / fingerprint tags / golden vectors) has silent traps and required manual steps. You MUST read `.github/instructions/projectconfig-fingerprint.instructions.md` before such changes. This also applies to CONSUMERS of fingerprint data (e.g., checking changed components between two commits via lockfile data).
+
 **IMPORTANT**: Code generation runs automatically with build/test commands. `mage generate` (runs `go generate` for each package in parallel) is a prerequisite for building and runs automatically with `mage build` and `mage unit`. `mage docs` rebuilds the binary and updates the JSON schema (`schemas/azldev.schema.json`) and CLI docs (`docs/user/reference/cli/`). Run `mage docs` explicitly after changing config structs or Cobra command descriptions so that checked-in generated files stay current (checked by PR gates).
 
 **CRITICAL**: Run `mage scenarioUpdate` when test expectations change (updates snapshots).

.github/instructions/go.instructions.md

@@ -98,19 +98,16 @@ description: "Instructions for working on the azldev Go codebase. IMPORTANT: Alw
     }
     ```
 
-## Component Fingerprinting — `fingerprint:"-"` Tags
+## Component fingerprinting & config structs
 
-Structs in `internal/projectconfig/` are hashed by `hashstructure.Hash()` to detect component changes. Fields **included by default** (safe: false positive > false negative).
+Editing structs in `internal/projectconfig/` or the fingerprint substrate in
+`internal/fingerprint/` - adding a field, touching `projectV1`, fingerprint tags,
+or golden vectors - has silent traps where a field vanishes or a frozen byte moves
+with the compiler still green. **Those rules live in
+[`projectconfig-fingerprint.instructions.md`](./projectconfig-fingerprint.instructions.md)**,
+read it before such changes.
 
-When adding a new field to a fingerprinted struct, ask: **"Does changing this field change the build output?"**
-- **Yes** (build flags, spec source, defines, etc.) → do nothing, included automatically.
-- **No** (human docs, scheduling hints, CI policy, metadata, back-references) → add `fingerprint:"-"` to the struct tag and register the exclusion in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
-
-If a parent struct field is already excluded (e.g. `Failure ComponentBuildFailureConfig ... fingerprint:"-"`), do **not** also tag the inner struct's fields — `hashstructure` skips the entire subtree.
-
-Run `mage unit` to verify — the guard test will catch unregistered exclusions or missing tags.
-
-### Cmdline Returns
+## Cmdline Returns
 
 CLI commands should return meaningful structured results. azldev has output formatting helpers to facilitate this (for example, `RunFunc*` wrappers handle formatting, so callers typically should not call `reflectable.FormatValue` directly).
 

.github/instructions/projectconfig-fingerprint.instructions.md

@@ -0,0 +1,133 @@
+---
+applyTo: "internal/projectconfig/**/*.go,internal/fingerprint/**/*.go"
+description: "Rules for interacting with project config structs and the fingerprint substrate. IMPORTANT: read before adding or changing any field on a config struct, or touching projectV1 / golden vectors / fingerprint tags."
+---
+
+# Config structs & fingerprinting
+
+Editing config structs (`internal/projectconfig/`) and the fingerprint substrate
+(`internal/fingerprint/`) has several **silent traps** - the compiler stays green
+while a field quietly vanishes or a frozen byte moves. Follow these rules.
+
+## `ComponentConfig.WithAbsolutePaths` hand-lists its fields
+
+`(*ComponentConfig).WithAbsolutePaths` in `internal/projectconfig/component.go`
+builds its result with an **explicit field-by-field struct literal**, not
+`deep.MustCopy`. It is called for *every* component at load time (via
+`mergeComponents` in `loader.go`), so any field missing from that literal is
+**silently zeroed on load** - it parses from TOML fine, then disappears before
+resolution, fingerprinting, or rendering. No error, no panic, just lost data.
+
+> It is hand-written (unlike every sibling `WithAbsolutePaths`, which uses
+> `deep.MustCopy`) for one reason: it must **alias** the cyclic, project-wide
+> `SourceConfigFile` pointer rather than deep-copy it. Do not "simplify" it to
+> a blanket `deep.MustCopy` - that reintroduces the cycle.
+
+**Every new field on `ComponentConfig` MUST be added to this literal.**
+
+## Adding a field to a config struct
+
+1. **Add the field** with a `toml:` key and a **mandatory** `fingerprint:` tag.
+   An absent tag fails `TestAllFingerprintedFieldsHaveDecision`
+   (`internal/projectconfig/fingerprint_test.go`).
+   - Build input → `fingerprint:"v1..*"` (measured).
+   - Not a build input → `fingerprint:"-"` **and** register the field in
+     `expectedExclusions` in that same test file. If the field's **parent struct is
+     already excluded** (e.g. `ComponentBuildConfig.Failure`), do *not* tag the
+     inner struct's fields - that subtree is pruned from both the decision walk and
+     `projectV1`, so its fields are neither policed nor measured.
+2. **If the field is on `ComponentConfig` directly → add it to
+   `WithAbsolutePaths`.** Skip this and it never loads.
+3. **If you added a new struct *type* to the fingerprinted graph**, add it to
+   `FingerprintedStructTypes()` in `internal/fingerprint/project.go` - the single
+   source of truth that **both** the mandatory-tag decision test
+   (`internal/projectconfig/fingerprint_test.go`) and the emission probe
+   (`measuredLeafEmitKeys` in `internal/fingerprint/project_internal_test.go`) walk.
+   Add it there and nowhere else.
+4. **If measured, emit it in `projectV1`** (`internal/fingerprint/project.go`,
+   hand-written as of v1) inside the correct sub-projector, under its **frozen `toml`
+   emit-key** (or an explicit `key=` in the tag - never the Go field name).
+   - Removing a measured field won't compile (good); a Go rename is byte-neutral
+     **only if** the `toml` key / `key=` is unchanged.
+5. **Set the field non-zero in `emissionProbeConfig()`**
+   (`internal/fingerprint/project_internal_test.go`). The emission probe FAILS
+   until you do - that is the guard against "measured but forgot to emit".
+6. **Append a `<toml-key>-set` golden vector** via
+   `go test ./internal/fingerprint -run TestGoldenVectors -update-golden`.
+7. **Regenerate the schema in all three places** if the field is user-facing (has
+   a `toml` key / `jsonschema` tag): `mage docs` updates
+   `schemas/azldev.schema.json` **and** the CLI reference docs, while
+   **`mage scenarioUpdate`** updates the `config generate-schema` scenario
+   snapshots (`TestSnapshots*config_generate-schema*`), which embed the schema
+   again. Running only `mage docs` leaves those snapshots red. Also hand-update the
+   relevant `docs/user/reference/config/` page (e.g. `components.md`).
+
+**Build-meaningful zero value?** A field whose *zero value* is itself a deliberate
+build setting (e.g. a `bool` where `false` carries meaning) is tagged
+`fingerprint:"!v1..*"` (always-emit) and wired with `builder.emitAlways(...)`, **not**
+`builder.emit(...)` - the omit-if-zero `emit` would drop it at its zero value and lose
+that signal. It MUST get a golden vector **at its zero value**; the
+differ-from-`minimal` guard in `golden_internal_test.go` then proves it actually emits.
+
+## Golden vectors: append-only, and the frozen/growing split
+
+- `maximalConfig()` is **FROZEN** - it is a golden-corpus config at the time of
+  the initial implementation of that fingerprint algorithm, so its digest is
+  pinned. **Never add a new field to `maximalConfig()`** (it would move the frozen
+  `maximal` digest, a hard CI failure). Field growth goes in
+  `emissionProbeConfig()` only.
+- Existing golden digests are **append-only**: `-update-golden` may add a new
+  `(name -> digest)` entry, but a CI check FATALs any commit that *modifies* an
+  existing digest. A moved frozen byte is a hard failure, not something a reviewer
+  must eyeball.
+- A new **omit-if-zero measured field that no config sets is drift-neutral**: it
+  emits no bytes, so no existing lock should move and no component version bump
+  is needed. Only a component that actually sets the field drifts.
+- **One isolation vector per new field is enough.** Add a single `<toml-key>-set`
+  vector that sets *only* the new field (like `single-overlay` /
+  `defines-empty-value`). Projection slots are length-prefixed and independent, so
+  a field's encoding is fully pinned in isolation - do **not** add a "maximal + new
+  field" or "previous + new field" combinatorial vector (and you cannot touch the
+  frozen `maximal`). Add *extra* vectors only for genuine internal discrimination
+  cases the field introduces (empty-vs-absent, map-key membership, etc.).
+
+## Other config struct / fingerprinting pitfalls
+
+- **Strict parsing.** The loader runs `DisallowUnknownFields()`
+  (`internal/projectconfig/loader.go`). Renaming or removing a `toml` key breaks
+  every existing on-disk config that still uses it. Load-time config migration is
+  deferred (lazy schema migration RFC PR E); until it exists, a `toml`-key rename
+  needs a coordinated, version-pinned rollout, not a bare rename.
+  (`fingerprint:"...,key=<old>"` keeps the *fingerprint* byte-neutral, but does
+  **not** keep old TOML parseable.)
+- **`omitempty` + go-toml/v2 (`v2.3.1`).** Emptiness is decided by reflecting the
+  **exported** fields *before* any `TextMarshaler` runs. A struct that marshals via
+  `TextMarshaler` but holds its real value in an **unexported** field is treated as
+  empty and **dropped even when set**. Keep round-tripped state in an exported field
+  or a plain string.
+- **`MergeUpdatesFrom` uses mergo with `WithOverride` + `WithAppendSlice`.** Slices
+  *append* across merge layers, and the same intent can resolve to `nil` *or* `[]`
+  depending on merge order. That is exactly why the projection's omit predicate
+  treats a nil-or-empty scalar slice as a zero value (both collapse to no emitted
+  bytes) at the hash boundary - do not rely on raw slice identity for a
+  fingerprinted field.
+- **A nested-struct field surfaces asymmetrically in output.** `encoding/json`
+  `omitempty` does **not** drop an empty struct, so `component list -O json` shows
+  `"foo":{}` on every component, while `config dump` (go-toml) *does* honor struct
+  `omitempty` and omits it. Neither is fingerprint drift (the projector omits on
+  projected emptiness) - don't mistake the JSON `{}` for churn.
+
+## Standing invariants (do not break)
+
+- **No reader recomputes a historical fingerprint.** Synthetic history and
+  historic-overlay application read **stored** lock strings only
+  (`synthistory.go`); only `computeCurrentFingerprint` (current tree) and the
+  `update` re-stamp call `ComputeIdentity`.
+- **Stored fingerprint is the atomic `v1:sha256:<digest>` token**; lock *format*
+  `version` stays `1` (the content version lives in the token prefix). A pre-`v1`
+  (prefix-less) token reconciles by **force-rehash** - the existing string
+  inequality re-stamps it; do not make that compare version-aware before the replay
+  registry (RFC PR C) exists.
+- **`projectV1` output is frozen.** A new byte encoding is a new content version,
+  never an edit to `projectV1`'s output. The golden vectors enforce this. The only
+  allowed changes to `projectV1` are purely additive.

docs/developer/reference/component-identity-and-locking.md

@@ -6,29 +6,41 @@
 - **Build optimization**: only rebuild changed components and their dependents, skipping unchanged ones.
 - **Automatic release bumping**: increment the release tag of changed packages automatically, and include the commits that caused the change in the changelog.
 
+> **Design & how-to.** The substrate design (version-set tags, the `v1:sha256:` content token, force-rehash reconciliation, and the future lazy schema migration) is specified in the [Lock-File Fingerprint Reset RFC](../rfc/lazy-schema-migration.md). The step-by-step rules for adding a fingerprinted field live in [`projectconfig-fingerprint.instructions.md`](../../../.github/instructions/projectconfig-fingerprint.instructions.md).
+
 ## Fingerprint Inputs
 
 A component's fingerprint is a SHA256 combining:
 
-1. **Config hash** — `hashstructure.Hash()` of the resolved `ComponentConfig` (after all merging). Fields tagged `fingerprint:"-"` are excluded.
-2. **Source identity** — content hash for local specs (all files in the spec directory), commit hash for upstream.
-3. **Overlay file hashes** — SHA256 of each file referenced by overlay `Source` fields.
-4. **Distro name + version**
-5. **Manual release bump counter** — increments with each manual release bump, ensuring a new fingerprint even if there are no config or source changes.
+1. **Config projection digest** - `sha256` of the canonical `projectV1` projection of the resolved `ComponentConfig` (after all merging). Only fields whose `fingerprint` tag measures them at v1 are emitted; `fingerprint:"-"` fields are excluded. A nil-or-empty scalar slice is treated as zero and omitted by the projection's omit predicate, so a merge-order nil-vs-`[]` difference never moves the digest.
+2. **Source identity** - content hash for local specs (all files in the spec directory), commit hash for upstream.
+3. **Overlay file hashes** - SHA256 of each file referenced by overlay `Source` fields.
+4. **Release version** - the distro's formal `releasever` (e.g. `4.0`), which feeds RPM macros like `%{dist}`; different release versions produce different package NEVRAs even with identical specs.
+5. **Manual release bump counter** - increments with each manual release bump, ensuring a new fingerprint even if there are no config or source changes.
+
+The combined digest is stored as a single self-describing **content-version token**: `input-fingerprint = "v1:sha256:<hex>"`. The `v1` prefix is the *content* version (the projection/encoding generation), written together with the digest so the two can never desync. It is independent of the lock *format* `version` field, which stays `1`. A pre-reset (prefix-less) token reads as sub-floor and is force-rehashed to `v1` on the next `component update`.
 
 Global change propagation works automatically: the fingerprint operates on the fully-merged config, so a change to a distro or group default changes the resolved config of every inheriting component.
 
-## `fingerprint:"-"` Tag System
+## `fingerprint` Version-Set Tags
+
+Every fingerprinted field carries an explicit `fingerprint` tag declaring **which content versions measure it**. There is no "included by default": an absent tag on a fingerprinted struct field fails the decision test (`TestAllFingerprintedFieldsHaveDecision`), so adding a field forces a conscious measure/exclude decision.
+
+Grammar (`internal/fingerprint/versiontag.go`):
 
-The `hashstructure` library uses `TagName: "fingerprint"`. Untagged fields are **included by default** (safe default: false positive > false negative).
+- `v1..*` - measured from v1 onward (the common case for a build input).
+- `vN..vM`, `vN..*`, comma-separated sets (`v1..v1,v3..*`) - measured only in those ranges.
+- `-` - never measured (excluded); register it in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
+- `!` prefix on a range (`!v1..*`) - always emit even at the zero value (scalar leaves only; an always-emit composite is rejected at v1).
+- `key=<name>` - override the emit-key (see below).
 
-A guard test (`TestAllFingerprintedFieldsHaveDecision`) reflects over all fingerprinted structs and maintains a bi-directional allowlist of exclusions. It fails if a `fingerprint:"-"` tag is added without registering it, or if a registered exclusion's tag is removed.
+**Emit-key is the frozen `toml:` key, never the Go field name.** `projectV1` emits each measured field under its `toml:` key (or an explicit `key=`), sorted by that key. So a cosmetic Go rename is byte-neutral, and a TOML-key rename stays byte-neutral by pinning `key=<old>`. Renaming the emit-key itself is an output change, and therefore a new content version.
 
 ### Adding a New Config Field
 
 1. Add the field to the struct in `internal/projectconfig/`.
-2. **If NOT a build input**: add `fingerprint:"-"` to the struct tag and register it in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
-3. **If a build input**: do nothing — included by default.
+2. Tag it: `fingerprint:"v1..*"` if it is a build input, or `fingerprint:"-"` (plus an `expectedExclusions` entry) if it is not.
+3. A measured, omit-if-zero field is **drift-neutral**: components that do not set it emit identical bytes, so no existing lock moves - it needs no content-version bump.
 4. Run `mage unit`.
 
 ### Adding a New Source Type
@@ -39,4 +51,4 @@ A guard test (`TestAllFingerprintedFieldsHaveDecision`) reflects over all finger
 
 ## Known Limitations
 
-- It is difficult to determine WHY a diff occurred (e.g., which specific field changed) since the fingerprint is a single opaque hash. The JSON output includes an `inputs` breakdown (`configHash`, `sourceIdentity`, `overlayFileHashes`, etc.) that can help narrow it down by comparing the two identity files manually.
+- It is difficult to determine WHY a diff occurred (e.g., which specific field changed) since the fingerprint is a single opaque `v1:sha256:` token: `ComponentIdentity` emits only the combined `fingerprint`, not a per-input breakdown.

go.mod

@@ -25,6 +25,7 @@ require (
 	github.com/go-playground/validator/v10 v10.30.3
 	github.com/google/renameio v1.0.1
 	github.com/google/uuid v1.6.0
+	github.com/gowebpki/jcs v1.0.1
 	github.com/h2non/gock v1.2.0
 	github.com/invopop/jsonschema v0.14.0
 	github.com/jedib0t/go-pretty/v6 v6.8.1
@@ -35,7 +36,6 @@ require (
 	github.com/magefile/mage v1.17.2
 	github.com/mark3labs/mcp-go v0.55.0
 	github.com/mattn/go-isatty v0.0.22
-	github.com/mitchellh/hashstructure/v2 v2.0.2
 	github.com/moby/moby/api v1.54.2
 	github.com/muesli/termenv v0.16.0
 	github.com/nxadm/tail v1.4.11

go.sum

@@ -169,6 +169,8 @@ github.com/google/renameio v1.0.1 h1:Lh/jXZmvZxb0BBeSY5VKEfidcbcbenKjZFzM/q0fSeU
 github.com/google/renameio v1.0.1/go.mod h1:t/HQoYBZSsWSNK35C6CO/TpPLDVWvxOHboWUAweKUpk=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gowebpki/jcs v1.0.1 h1:Qjzg8EOkrOTuWP7DqQ1FbYtcpEbeTzUoTN9bptp8FOU=
+github.com/gowebpki/jcs v1.0.1/go.mod h1:CID1cNZ+sHp1CCpAR8mPf6QRtagFBgPJE0FCUQ6+BrI=
 github.com/h2non/gock v1.2.0 h1:K6ol8rfrRkUOefooBC8elXoaNGYkpp7y2qcxGG6BzUE=
 github.com/h2non/gock v1.2.0/go.mod h1:tNhoxHYW2W42cYkYb1WqzdbYIieALC99kpYr7rH/BQk=
 github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 h1:2VTzZjLZBgl62/EtslCrtky5vbi9dd7HrQPQIx6wqiw=
@@ -228,8 +230,6 @@ github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh
 github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
 github.com/mattn/go-runewidth v0.0.21 h1:jJKAZiQH+2mIinzCJIaIG9Be1+0NR+5sz/lYEEjdM8w=
 github.com/mattn/go-runewidth v0.0.21/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
-github.com/mitchellh/hashstructure/v2 v2.0.2 h1:vGKWl0YJqUNxE8d+h8f6NJLcCJrgbhC4NcD46KavDd4=
-github.com/mitchellh/hashstructure/v2 v2.0.2/go.mod h1:MG3aRVU/N29oo/V/IhBX8GR/zz4kQkprJgF2EVszyDE=
 github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
 github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
 github.com/moby/go-archive v0.2.0 h1:zg5QDUM2mi0JIM9fdQZWC7U8+2ZfixfTYoHL7rWUcP8=