chore(comps): annotate backport overlays with category metadata#17727
chore(comps): annotate backport overlays with category metadata#17727liunan-ms wants to merge 2 commits into
Conversation
| replacement = "" | ||
| # Overlays live as one-logical-change-per-file under ./overlays/. | ||
| # See https://github.com/microsoft/azure-linux-dev-tools/blob/main/docs/user/reference/config/overlays.md#per-file-overlay-format-overlaytoml | ||
| overlay-dir = "overlays" |
There was a problem hiding this comment.
It would be great to update the AI instructions to explain when to use this pattern vs. not, and what sort of data to add to each overlay.
There was a problem hiding this comment.
Yes, I'll add a SKILL.md and instruction in a separate PR for using this pattern and suggesting annotations for new overlays.
liunan-ms
changed the title
liunan-ms
force-pushed
the
liunan/overlay_metadata
branch
from
93c3e9b to
e29cdb5
Compare
liunan-ms
force-pushed
the
liunan/overlay_metadata
branch
from
23ed86c to
4107a39
Compare
There was a problem hiding this comment.
Pull request overview
This PR annotates the Azure Linux overlays previously flagged by the overlay classifier with structured metadata (a category plus upstream commits/pr/bugs), so future Fedora rebases can mechanically identify and retire backported overlays once they land upstream. It also bumps the pinned azldev version to one that understands these annotations and a new per-file overlay layout.
Changes:
- Inline annotations (17 components): add a
[components.<name>.overlays.metadata]block to existing single-/few-overlay.comp.tomlfiles. Note these attach per-overlay (to the immediately preceding overlay), which is used deliberately in some files (e.g.sos,sssd) but is inconsistent inxbean. - Per-file migration (7 components):
cpio,gnulib,javapackages-tools,qemu,zbar,apache-ivy,cloud-initmove inline overlays intooverlays/*.overlay.tomlfiles (top-level[metadata]+[[overlays]], withsourcepaths rewritten to../to point back at the component root).qemualso consolidates per-section subpackage removals intospec-remove-subpackage(verified output-equivalent against the rendered spec). .azldev-versionbump to enable the newoverlay-files/metadatafeatures; the committed authoritative schema was not regenerated to match.
Reviewed changes
Copilot reviewed 39 out of 39 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
.azldev-version |
Bumps pinned azldev to a version supporting overlay-files/metadata; committed schema not regenerated (flagged). |
base/comps/{ant,atlas,glade,gt,intel-qpl,maven,maven4,xclock,xmvn,xmvn5}/*.comp.toml |
Single-overlay inline metadata annotations (correct attachment). |
base/comps/{sos,sssd,kdump-utils,rust-podman-sequoia,vamp-plugin-sdk}/*.comp.toml |
Multi-overlay inline metadata with distinct per-overlay commits (correct). |
base/comps/xbean/xbean.comp.toml |
Inline metadata; remove/add openjdk retarget pair annotated inconsistently (flagged). |
base/comps/cpio/* |
Migrated to per-file overlay; rmt BR drop annotated. |
base/comps/gnulib/* |
Migrated; temporary Source-URL backport annotated. |
base/comps/javapackages-tools/* |
Migrated; openjdk21-drop multi-commit backport consolidated. |
base/comps/qemu/* |
Migrated to 4 overlay files; subpackage removals consolidated to spec-remove-subpackage (output-equivalent). |
base/comps/zbar/* |
Migrated; GIR-drop (backport) and video-disable (pruning) split into two files. |
base/comps/apache-ivy/* |
Migrated; 0001 overlays lack required description fields (flagged nit). |
base/comps/cloud-init/* |
Migrated to 3 overlay files (cpe-fix, azl4 patch series, branding); ../ source paths point at component-root patches. |
liunan-ms
force-pushed
the
liunan/overlay_metadata
branch
from
4107a39 to
13381f7
Compare
- apache-ivy: add per-overlay descriptions to the pack200 backport file (hygiene rule #1 — every overlay must explain why the change is needed). - gnulib: drop the redundant per-component overlay-files declaration; the project-wide [default-component-config] already discovers overlays/*.overlay.toml. - grub2: reconcile the os-prober revert overlays (0002 + 0003) to a single category. Re-disabling os-prober restores upstream GRUB's secure default that Fedora diverged from (it will not retire on a Fedora rebase), so both halves are azl-security-compliance and the misleading backport commits are dropped. - external/schemas/azldev.schema.json: regenerate against the pinned azldev (4b5f72c) so the authoritative schema knows the new metadata / overlay-files / BugRef fields used throughout this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
liunan-ms
force-pushed
the
liunan/overlay_metadata
branch
from
f55c1ce to
7db736c
Compare
| # ant-openjdk21 subpackage has an unsatisfiable dependency. | ||
| [metadata] | ||
| category = "backport-dist-git" | ||
| upstreamable = false |
There was a problem hiding this comment.
For something that was upstreamed, what do we want to say for upstreamable?
Our non-upstreamable changes are debt that we carry. A backported patch isn't really the same kind of debt.
Makes me wonder if upstream-status should have been an enum, e.g.: inapplicable vs. upstream vs. not-started.
Without that we'll presumably need to look for non-upstreamable changes and also filter out certain categories, which seems messier.
There was a problem hiding this comment.
I agree that the semantics of upstreamable on a backport are kind of awkward. Here is how we can read the current field:
upstreamable = false:
- backport fixes that're already upstreamed
- azl-only changes that should not be upstreamed
upstreamable = true:
- the change should be upstreamed
Expanding upstreamable into an enum will help differentiate the 2 cases while upstreamable = false, but in practice should we more care about the change should be upstreamed by looking for upstreamable = true because these are actional work? For backport fixes that are already upstreamed, we should remove the overlay once pinning to a new upstream version and azl-only changes will stay untouched.
There was a problem hiding this comment.
Addressed in commit 64bcb0f — the upstreamable boolean has been replaced by an upstream-status enum as you suggested. Backported patches that already landed Fedora upstream now use upstream-status = "upstreamed", cleanly distinguishing the following ones and AZL-specific changes (inapplicable):
- upstreamable: the patch we're carrying is itself upstream-shaped. Sending the exact same diff (or something very close) to Fedora / the upstream project would be accepted. This status is used for the overlays that is available in OSS project upstream (either released or in an open PR) but not in Fedora upstream yet and the local patches that we created and should be pushed to both OSS project and Fedora upstream. To differentiate the two cases in the existing overlays, the first one always has upstream commits in its metadata, the later one doesn't have any upstream commit yet. For new overlays, it's highly recommended to push the local fixes to OSS upstream first and grab the commit into the overlay metadata.
- needs-upstream-hook: the change is AZL-specific and would not be accepted upstream as-is (e.g. hard-coding
Vendor: Microsoft, disabling a feature only we don't want, swapping a default path). But upstream could add abcond / %if / configknob that would let us express the same customization without patching the spec.
The schema has been regenerated to match.
| # unsatisfiable. Retarget to the openjdk25 variant. | ||
| [metadata] | ||
| category = "azl-pruning" | ||
| upstreamable = false |
There was a problem hiding this comment.
Upstream will need to move to 25, so I think it is.
There was a problem hiding this comment.
now uses upstream-status = "upstreamable"
| { description = "Add BuildRequires on ant-openjdk25 in place of ant-openjdk21", type = "spec-add-tag", tag = "BuildRequires", value = "ant-openjdk25" }, | ||
| { description = "Retarget JAVA_HOME from java-21-openjdk to java-25-openjdk in %build", type = "spec-search-replace", section = "%build", regex = "java-21-openjdk", replacement = "java-25-openjdk" }, | ||
| ] | ||
| # Overlays live as one-logical-change-per-file under ./overlays/, discovered via |
There was a problem hiding this comment.
Does this comment need to be here?
There was a problem hiding this comment.
The comment is removed in all comp.toml.
| type = "spec-search-replace" | ||
| regex = 'liblapack_pic\.a' | ||
| replacement = "liblapack.a" | ||
| # Overlays live as one-logical-change-per-file under ./overlays/, discovered via |
| type = "spec-search-replace" | ||
| regex = 'python3 tools/render-template --variant %{\?rhel:rhel}%{!\?rhel:fedora}' | ||
| replacement = "python3 tools/render-template --variant azurelinux" | ||
| # Overlays live as one-logical-change-per-file under ./overlays/, discovered via |
| # platform.m4 sourcing /etc/os-release overwriting the VERSION shell variable. | ||
| [metadata] | ||
| category = "backport-dist-git" | ||
| upstreamable = false |
There was a problem hiding this comment.
Why do we sometimes omit this and sometimes explicitly mark it false? Which is preferred?
There was a problem hiding this comment.
I explicitly mark the overlays that already have upstream fixes as upstreamable = false, omit it for the unsure ones, leaving them for future revisit. For example, fedora upstream has a patch to fix the same issue which is fixed by the overlays (not using the patch) in gt, so I'm not sure if this is upstreamable or even if this overlay should be a backport or not.
There was a problem hiding this comment.
Now upstream-status is a required field, which will push the overlay contributor to pick a definite status from upstreamed, upstreamable, needs-upstream-hook, inapplicable, or unknown for the change.
| # Backport of upstream SSSD PR sssd/sssd#8397 (merged as 308af8f2): fix | ||
| # platform.m4 sourcing /etc/os-release overwriting the VERSION shell variable. | ||
| [metadata] | ||
| category = "backport-dist-git" |
There was a problem hiding this comment.
Also not a backport from dist-git.
There was a problem hiding this comment.
The category was renamed from backport-dist-git to upstream-backport, which is more accurate and covers backports from both the upstream project and dist-git.
| # Mirrors the fix in Fedora rawhide vamp-plugin-sdk-2.10-4; the Makefile race | ||
| # fix is upstream c4dm/vamp-plugin-sdk commit da2d66f. | ||
| [metadata] | ||
| category = "backport-dist-git" |
| # Backport fix for ARM64 SVE static assertion failure from upstream commit 0fd437f. | ||
| # Changes std::is_same to std::is_base_of to handle derived arch types like i8mm<neon64>. | ||
| [metadata] | ||
| category = "backport-dist-git" |
| # documents without repeating `overlay-files` in every comp.toml. | ||
| # See https://github.com/microsoft/azure-linux-dev-tools/blob/main/docs/user/reference/config/overlays.md#per-file-overlay-format | ||
| [default-component-config] | ||
| overlay-files = ["overlays/*.overlay.toml"] |
There was a problem hiding this comment.
The other PR has this in the distro config. Where do we want it?
There was a problem hiding this comment.
I think components.toml is the best place to define overlay-files in this PR. The overlay-files defined in distro config from the other PRs are temporarily for PR check, will be moved before PR merge.
liunan-ms
force-pushed
the
liunan/overlay_metadata
branch
from
7db736c to
cdba584
Compare
liunan-ms
force-pushed
the
liunan/overlay_metadata
branch
from
cdba584 to
74797a7
Compare
Pin azldev to 830e6fed79750f8e24ae17cd630248f50c90c8a9, which builds on the upstream-backport overlay category, URLRef (replacing BugRef), and required upstream-status metadata (replacing the old upstreamable boolean). It also renames the azl-dep-missing-workaround category to azl-temp-workaround and adds an optional [metadata] block to component groups. Regenerate external/schemas/azldev.schema.json against the pinned tool so the authoritative schema matches the new overlay metadata model, including upstream-status, the renamed azl-temp-workaround category, and component-group metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Annotate every overlay surfaced by the overlay classifier with structured [metadata] so future Fedora rebases can mechanically identify and retire overlays that have landed upstream. - Add a project-wide [default-component-config] in base/comps/components.toml with overlay-files = ["overlays/*.overlay.toml"], so each component discovers its own per-file overlays without redeclaring the glob. - Migrate all 24 annotated components to the per-file overlay-files layout: inline overlays move into per-logical-change overlays/*.overlay.toml files, each carrying one file-level [metadata] block. Patch source paths are rewritten to ../<patch> to point back at the component root. - Categories: most overlays are upstream-backport (with verified commits); a few non-backport overlays are labelled accordingly (azl-pruning, azl-security-compliance, azl-compatibility, azl-branding-policy). Pure annotation/restructuring: the rendered spec is byte-identical before and after (verified via prepare-sources), and azldev comp update reports no lock drift (metadata is excluded from fingerprints).
liunan-ms
force-pushed
the
liunan/overlay_metadata
branch
from
74797a7 to
3414562
Compare
This PR annotates the overlays surfaced by the overlay classifier with structured
[metadata](acategoryplus, for genuine backports, the verified upstreamcommits), so future Fedora rebases can mechanically identify and retire overlays that have landed upstream. It also pinsazldevto a version that understands these annotations and the per-fileoverlay-fileslayout, and regenerates the authoritative JSON schema to match.Changes:
Project config
base/comps/components.tomlgains a project-wide[default-component-config]withoverlay-files = ["overlays/*.overlay.toml"], so every component discovers its ownoverlays/*.overlay.tomlwithout redeclaring the glob.Per-file overlay migration (24 components)
ant, apache-ivy, atlas, cloud-init, cpio, glade, gnulib, grub2, gt, intel-qpl, javapackages-tools, kdump-utils, maven, maven4, rust-podman-sequoia, sos, sssd, vamp-plugin-sdk, xbean, xclock, xmvn, xmvn5, xsimd, zbar — inline overlays were moved into per-logical-change
overlays/*.overlay.tomlfiles, each carrying one file-level[metadata]block. Multi-commit backports (e.g.javapackages-toolsopenjdk21 drop) consolidate related upstream commits into a single file viacommits = [url1, url2, …]. Patchsourcepaths are rewritten to../<patch>to point back at the component root.Category spread
Most overlays are
upstream-backport(with verified upstreamcommits). A few non-backport overlays surfaced alongside them are labelled accordingly:azl-pruning(e.g. grub2 xen-module disable),azl-security-compliance(grub2 os-prober reverts — restoring upstream GRUB's secure default Fedora diverged from),azl-compatibility, andazl-branding-policy.azldev changes:
0256227→830e6feNew features
azldev component history— new CLI command showing per-component change history (Remove toolchain-local-wget-list after use #212).azldev repo query— new command for inspecting/managing RPM repositories (Update libestr #213).[metadata]block on component overlays, then reworked to typedURLReflist +upstream-statusenum; categoryazl-dep-missing-workaroundrenamed toazl-temp-workaround.[component-groups.<name>.metadata]sharing the overlay-metadata schema.overlay-filesfield +.overlay.tomlper-file format; fixed to expand after config resolution.Tooling / build
mage mutation+ MCP (Switching to correct source for the Microsoft cert bundle. #244).actions/checkout,golang.org/x/net, andgo-tomldependency bumps.Validation
prepare-sourcesdiffs for every migrated component).azldev comp updatereports no lock drift (metadata is excluded from fingerprints; theoverlay-filesrestructuring keeps the same input set).azldev comp renderall passed in PR check.