feat(overlays): add overlay metadata and per-file overlay format (1/3) by liunan-ms · Pull Request #249 · microsoft/azure-linux-dev-tools

liunan-ms · 2026-06-24T05:10:06Z

This PR is the first of three-split out from the previous overlay-metadata PR (#242). It adds two related ways to document overlays without affecting how they're applied. The follow-ups will add component-group metadata (PR 2) and the azldev component metadata command that consumes both (PR 3).

What's new

Inline [metadata] block on overlays (commit 1)
Optional table on any overlay with:
- category (required) — one of backport-dist-git, azl-pruning, azl-compatibility, azl-branding-policy, azl-dep-missing-workaround, azl-disable-flaky-tests, azl-disable-unsupported-tests, azl-security-compliance, azl-release-management, azl-platform-adaptation
- commits — upstream commit URLs (required when category = "backport-dist-git")
- bugs — bug-tracker URLs
- upstreamable — true/false; omit when not yet assessed
Per-file overlay format via overlay-files (commit 2)
For logical changes that span multiple overlays (CVE backports, feature disablements, etc.), set overlay-files on a component to one or more globs. Each matched file declares a single file-level [metadata] block (stamped onto every overlay in the file) plus an ordered list of [[overlays]].
Files are loaded in filename order within each glob, globs concatenated in declaration order, duplicates de-duped. Per-overlay [metadata] is rejected inside an overlay file (file-level metadata is the single source of truth). Relative source paths resolve against the overlay file's directory. An empty glob match is a configuration error.

Notes

Metadata and overlay-files are excluded from component fingerprints — editing them never invalidates build caches.
Regenerated JSON schema and scenario snapshots.
No CLI surface yet; azldev component metadata consumes this in a follow-up PR.

Add an optional [metadata] block on overlays (category, commits, bug links, upstreamable) declared inline in component config files. The metadata is documentation only and is excluded from component fingerprints, so editing it never invalidates build caches. Regenerate JSON schema; update reference docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR adds documentation-only overlay metadata plus a new “per-file overlay” document format (overlay-files) that lets a component load and append overlays from one or more TOML files, without affecting overlay application behavior or component fingerprints.

Changes:

Added optional per-overlay metadata (category/commits/bugs/upstreamable) and validation for it.
Added overlay-files on components and a loader that expands globs, parses overlay documents deterministically, stamps file-level metadata onto overlays, and resolves source relative to the overlay file.
Updated fingerprint exclusions, config reference docs, JSON schema, and scenario snapshots.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
schemas/azldev.schema.json	Schema updates for `overlay-files` and overlay `metadata`.
scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap	Snapshot update reflecting regenerated schema output.
scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap	Snapshot update reflecting regenerated schema output.
internal/projectconfig/overlay.go	Adds `ComponentOverlay.Metadata` and validates it during overlay validation.
internal/projectconfig/overlay_metadata.go	Introduces `OverlayMetadata`/`BugRef`, category enum, cloning, and validation.
internal/projectconfig/overlay_metadata_test.go	Unit tests for metadata validation and overlay validation integration.
internal/projectconfig/overlay_file.go	Implements overlay-file parsing/loading, glob expansion, and metadata stamping.
internal/projectconfig/overlay_file_test.go	Unit tests for overlay-file loading behavior and ordering/deduplication.
internal/projectconfig/loader.go	Hooks overlay-file processing into config load before validation.
internal/projectconfig/fingerprint_test.go	Registers new fingerprint exclusions for overlay metadata and `overlay-files`.
internal/projectconfig/component.go	Adds `OverlayFiles` field and preserves it in `WithAbsolutePaths`.
docs/user/reference/config/overlays.md	Documents overlay metadata and the per-file overlay format.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

dmcilvaney · 2026-06-25T00:19:19Z

+			return nil, fmt.Errorf("%w: %q", ErrOverlayFilesNoMatches, pattern)
+		}
+
+		sort.Strings(matches)


Not sure it matters, all files should have the same directory structure, but as written in the docs it should sort by basename.

Thanks for catching this! Updated to align with the docs, sort files in filename lexicographic order.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Add ComponentConfig.OverlayFiles: a list of globs (relative to the component config file) that are matched at load time to discover per-file overlay documents. Each file declares a single [metadata] block plus an ordered list of [[overlays]]; the file-level metadata is stamped onto every overlay in the file. Matches are concatenated in declaration order and applied in filename order within each glob. Globs that match no files are surfaced as a configuration error. Per-overlay [metadata] is rejected inside an overlay file (the file-level metadata is the single source of truth). Relative source paths in overlay files are resolved against the overlay file's directory. OverlayFiles is excluded from the fingerprint because it affects only where overlays are sourced from, not their content. Regenerate JSON schema; update reference docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

reubeno

Generally looks pretty good to me -- just a few questions/comments.

reubeno · 2026-06-25T21:39:51Z

+	// Resolve per-component overlay file globs, stamping each
+	// file's [metadata] onto its overlays and appending them to the component before
+	// validation runs.
+	if err := applyOverlayFiles(fs, cfg, permissiveConfigParsing); err != nil {


This seems out-of-place as a special case in the toplevel loader function. Is there a way we could have done this closer to where the component / overlays are being resolved/loaded?

I considered moving this closer to component resolution, but it would require passing permissiveConfigParsing into the merge/resolve path, which currently only works on already-decoded config structs. Daniel will clean up the loading stuff, and we can move this closer to component resolution then.

I think it's worth making that change -- I see this as adding debt; I'm okay if you defer to a later PR, but my request is that you file a bug on it and assign to yourself ( or Daniel if he's willing to take it) to make sure this doesn't get forgotten.

Copilot AI review requested due to automatic review settings June 24, 2026 05:10

Copilot started reviewing on behalf of liunan-ms June 24, 2026 05:10 View session

liunan-ms force-pushed the liunan/overlay-metadata-and-files branch from d38d17d to cf624a1 Compare June 24, 2026 05:10

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread internal/projectconfig/overlay_metadata.go

Comment thread internal/projectconfig/overlay_file.go

Comment thread internal/projectconfig/overlay_file.go Outdated

Copilot AI review requested due to automatic review settings June 24, 2026 16:57

liunan-ms force-pushed the liunan/overlay-metadata-and-files branch from cf624a1 to be5450a Compare June 24, 2026 16:57

Copilot started reviewing on behalf of liunan-ms June 24, 2026 16:58 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread internal/projectconfig/overlay.go Outdated

liunan-ms force-pushed the liunan/overlay-metadata-and-files branch from be5450a to 9890d21 Compare June 24, 2026 17:26

liunan-ms changed the title ~~feat(overlays): add overlay metadata and per-file overlay format~~ feat(overlays): add overlay metadata and per-file overlay format (1/3) Jun 24, 2026

dmcilvaney reviewed Jun 25, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 25, 2026 16:54

liunan-ms force-pushed the liunan/overlay-metadata-and-files branch from 9890d21 to 64b2ed9 Compare June 25, 2026 16:54

Copilot started reviewing on behalf of liunan-ms June 25, 2026 16:55 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread internal/projectconfig/component.go Outdated

Comment thread internal/projectconfig/overlay_file.go

liunan-ms force-pushed the liunan/overlay-metadata-and-files branch from 64b2ed9 to 1516af2 Compare June 25, 2026 18:00

reubeno reviewed Jun 25, 2026

View reviewed changes

reubeno approved these changes Jun 25, 2026

View reviewed changes

liunan-ms mentioned this pull request Jun 26, 2026

Move overlay-files expansion closer to component resolution #253

Open

Merge branch 'main' into liunan/overlay-metadata-and-files

c4c5c36

liunan-ms merged commit 0b80263 into microsoft:main Jun 26, 2026
18 checks passed

liunan-ms deleted the liunan/overlay-metadata-and-files branch June 26, 2026 00:54

Uh oh!

Conversation

liunan-ms commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

dmcilvaney Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

liunan-ms Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

reubeno left a comment

Choose a reason for hiding this comment

Uh oh!

reubeno Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

liunan-ms Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

reubeno Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liunan-ms commented Jun 24, 2026 •

edited

Loading