diff --git a/docs/user/reference/config/components.md b/docs/user/reference/config/components.md index 35344a8d..33e984f4 100644 --- a/docs/user/reference/config/components.md +++ b/docs/user/reference/config/components.md @@ -11,7 +11,7 @@ A component definition tells azldev where to find the spec file, how to customiz | Spec source | `spec` | [SpecSource](#spec-source) | No | Where to find the spec file for this component. Inherited from distro defaults if not specified. | | Release config | `release` | [ReleaseConfig](#release-configuration) | No | Controls how the Release tag is managed during rendering | | Overlays | `overlays` | array of [Overlay](overlays.md) | No | Modifications to apply to the spec and/or source files | -| Overlay files | `overlay-files` | array of string | No | Glob patterns that load per-file overlay documents. See [Per-file overlay format](overlays.md#per-file-overlay-format). | +| Overlay files | `overlay-files` | array of string | No | Path or glob patterns that load per-file overlay documents after component config resolution. Inherited relative patterns are resolved from the concrete component's config file, or from the matched spec file's directory for spec-discovered components. Use `[]` to disable inherited patterns. See [Per-file overlay format](overlays.md#per-file-overlay-format). | | Build config | `build` | [BuildConfig](#build-configuration) | No | Build-time options (macros, conditionals, check config) | | Render config | `render` | [RenderConfig](#render-configuration) | No | Options controlling spec rendering behavior | | Source files | `source-files` | array of [SourceFileReference](#source-file-references) | No | Additional source files to download for this component | diff --git a/docs/user/reference/config/overlays.md b/docs/user/reference/config/overlays.md index 9a3abc9f..f635bef9 100644 --- a/docs/user/reference/config/overlays.md +++ b/docs/user/reference/config/overlays.md @@ -72,7 +72,7 @@ In addition to per-overlay fields, the following fields are set directly on the | Field | TOML Key | Description | |-------|----------|-------------| -| Overlay files | `overlay-files` | List of glob patterns (relative to the component's config file) matched against the filesystem at load time to locate per-file overlay documents. Patterns support `**` (globstar). Matches are concatenated in declaration order; within a single pattern, matches are applied in filename (lexicographic) order, with full path as a tie-breaker for duplicate filenames. Each pattern must match at least one file. Duplicate matches across patterns are de-duplicated. See [Per-file overlay format](#per-file-overlay-format). | +| Overlay files | `overlay-files` | List of path or glob patterns matched against the filesystem after component config resolution to locate per-file overlay documents. Relative patterns are resolved from the concrete component's config file, or from the matched spec file's directory for spec-discovered components. Patterns support `**` (globstar). Matches are concatenated in declaration order; within a single pattern, matches are applied in filename (lexicographic) order, with full path as a tie-breaker for duplicate filenames. Glob patterns that match no files are ignored; literal paths must match a file. Duplicate matches across patterns are de-duplicated. See [Per-file overlay format](#per-file-overlay-format). | ## Overlay Metadata @@ -163,6 +163,8 @@ When a single logical change (a CVE backport, a feature disablement, a Fedora ch Set `overlay-files` on the component to one or more globs (relative to the component config) and drop one overlay file per logical change into a directory of your choosing. The conventional layout uses a sibling `overlays/` directory and a `*.overlay.toml` filename suffix, but neither is required — `overlay-files` is just a glob, so any layout you can describe with `**`/`*` patterns works. +`overlay-files` can also be inherited from `default-component-config` at the project, distro, or component-group level. Inherited relative patterns are still resolved for each concrete component: from its component config file when it has one, or from the matched spec file's directory when it is discovered by a component group's `specs` pattern. This makes defaults useful for component-local discovery patterns such as `overlay-files = ["overlays/*.overlay.toml"]`. If a component sets `overlay-files`, that value replaces the inherited list; use `overlay-files = []` to disable inherited overlay files for a component, or include both patterns explicitly when you want to keep default discovery and add component-specific locations. + ``` base/comps/mypackage/ ├── mypackage.comp.toml @@ -178,7 +180,7 @@ base/comps/mypackage/ overlay-files = ["overlays/*.overlay.toml"] ``` -Files are loaded in **filename (lexicographic) order** within each glob, using the full path as a tie-breaker when multiple matches have the same filename. Globs are concatenated in declaration order, so prefix each file with a numeric ordinal (`0001-`, `0002-`, …) to make the apply order obvious and stable. Files that don't match any of your globs are ignored, so you can keep `README.md` or other notes alongside without naming them out explicitly. Each declared glob must match at least one file; an empty match is treated as a misconfiguration and surfaced as an error. +Files are loaded in **filename (lexicographic) order** within each glob, using the full path as a tie-breaker when multiple matches have the same filename. Globs are concatenated in declaration order, so prefix each file with a numeric ordinal (`0001-`, `0002-`, …) to make the apply order obvious and stable. Files that don't match any of your globs are ignored, so you can keep `README.md` or other notes alongside without naming them out explicitly. A declared glob that matches no files contributes no overlays; a literal path without wildcard characters must match a file. Overlays loaded via `overlay-files` are **appended after** any inline overlays declared directly on the component. diff --git a/internal/app/azldev/core/components/resolver.go b/internal/app/azldev/core/components/resolver.go index 5001ac1b..fad536a4 100644 --- a/internal/app/azldev/core/components/resolver.go +++ b/internal/app/azldev/core/components/resolver.go @@ -9,6 +9,7 @@ import ( "log/slog" "path" "path/filepath" + "slices" "strings" "github.com/bmatcuk/doublestar/v4" @@ -129,7 +130,9 @@ func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { var component Component // Resolve the config for this group member. - component, err = r.getComponentFromNameAndSpecPath(groupMember.ComponentName, groupMember.SpecPath) + component, err = r.getComponentFromNameAndSpecPath( + groupMember.ComponentName, groupMember.SpecPath, []string{groupName}, + ) if err != nil { return components, fmt.Errorf("failed to enumerate components in group '%s':\n%w", groupName, err) } @@ -148,7 +151,9 @@ func (r *Resolver) FindAllComponents() (components *ComponentSet, err error) { var updatedComponentConfig *projectconfig.ComponentConfig // Apply defaults from the loaded distro config... - updatedComponentConfig, err = applyInheritedDefaultsToComponent(r.env, componentConfig) + updatedComponentConfig, err = applyInheritedDefaultsToComponent( + r.env, componentConfig, overlayFilesReferenceDir(componentConfig, ""), nil, + ) if err != nil { return components, err } @@ -343,6 +348,48 @@ func componentGroupExcludesSpec( return false, nil } +func componentGroupMatchesSpecPath( + group *projectconfig.ComponentGroupConfig, specPath string, +) (matches bool, err error) { + for _, pattern := range group.SpecPathPatterns { + matched, err := doublestar.PathMatch(pattern, specPath) + if err != nil { + return false, fmt.Errorf( + "failed to compare %#q against spec pattern %#q:\n%w", specPath, pattern, err) + } + + if !matched { + continue + } + + excludes, err := componentGroupExcludesSpec(group, specPath) + if err != nil { + return false, err + } + + return !excludes, nil + } + + return false, nil +} + +func componentGroupNamesForSpecPath(env *azldev.Env, specPath string) ([]string, error) { + var groupNames []string + + for groupName, group := range env.Config().ComponentGroups { + matches, err := componentGroupMatchesSpecPath(&group, specPath) + if err != nil { + return nil, err + } + + if matches { + groupNames = append(groupNames, groupName) + } + } + + return groupNames, nil +} + func (r *Resolver) addComponentsBySpecPathToSet(specPath string, components *ComponentSet) error { var component Component @@ -395,7 +442,9 @@ func (r *Resolver) addComponentsByGroupNameToSet(groupName string, components *C for _, groupMember := range componentGroup.Components { var component Component - component, err = r.getComponentFromNameAndSpecPath(groupMember.ComponentName, groupMember.SpecPath) + component, err = r.getComponentFromNameAndSpecPath( + groupMember.ComponentName, groupMember.SpecPath, []string{groupName}, + ) if err != nil { return fmt.Errorf( "failed to enumerate components in group '%s':\n%w", groupName, err) @@ -421,7 +470,12 @@ func (r *Resolver) getComponentForSpecPath(specPath string) (component Component return component, fmt.Errorf("failed to verify spec '%s' exists:\n%w", specPath, statErr) } - return r.getComponentFromNameAndSpecPath(name, specPath) + groupNames, err := componentGroupNamesForSpecPath(r.env, specPath) + if err != nil { + return component, err + } + + return r.getComponentFromNameAndSpecPath(name, specPath, groupNames) } // Given a path to a .spec file, deduce the component's name. @@ -438,7 +492,9 @@ func deduceComponentNameFromSpec(specPath string) string { } // Finds the named component in the provided environment; returns its configuration. Returns error if it can't be found. -func (r *Resolver) getComponentFromNameAndSpecPath(name, specPath string) (component Component, err error) { +func (r *Resolver) getComponentFromNameAndSpecPath( + name, specPath string, groupNames []string, +) (component Component, err error) { config := r.env.Config() if config == nil { return component, errors.New("no project config loaded") @@ -459,7 +515,11 @@ func (r *Resolver) getComponentFromNameAndSpecPath(name, specPath string) (compo var updatedComponentConfig *projectconfig.ComponentConfig // Apply inherited defaults to the component. - updatedComponentConfig, err = applyInheritedDefaultsToComponent(r.env, foundComponentConfig) + overlayReferenceDir := overlayFilesReferenceDir(foundComponentConfig, specPath) + + updatedComponentConfig, err = applyInheritedDefaultsToComponent( + r.env, foundComponentConfig, overlayReferenceDir, groupNames, + ) if err != nil { return component, err } @@ -777,15 +837,27 @@ func (r *Resolver) warnOnLockDrift(resolved *ComponentSet) { } // Given an explicit component config, apply all inherited defaults. +func overlayFilesReferenceDir(component projectconfig.ComponentConfig, specPath string) string { + if component.SourceConfigFile != nil { + return component.SourceConfigFile.Dir() + } + + if specPath != "" { + return filepath.Dir(specPath) + } + + return "" +} + func applyInheritedDefaultsToComponent( - env *azldev.Env, component projectconfig.ComponentConfig, + env *azldev.Env, component projectconfig.ComponentConfig, overlayFilesReferenceDir string, extraGroupNames []string, ) (result *projectconfig.ComponentConfig, err error) { _, distroVer, err := env.Distro() if err != nil { return nil, fmt.Errorf("failed to resolve current distro:\n%w", err) } - groupNames := env.Config().GroupsByComponent[component.Name] + groupNames := componentGroupNames(env, component.Name, extraGroupNames) resolved, err := projectconfig.ResolveComponentConfig( component, @@ -798,9 +870,28 @@ func applyInheritedDefaultsToComponent( return nil, fmt.Errorf("resolving config for component '%s':\n%w", component.Name, err) } + resolved, err = projectconfig.ExpandResolvedOverlayFiles( + env.FS(), resolved, overlayFilesReferenceDir, env.PermissiveConfigParsing(), + ) + if err != nil { + return nil, fmt.Errorf("resolving overlay files for component '%s':\n%w", component.Name, err) + } + return &resolved, nil } +func componentGroupNames(env *azldev.Env, componentName string, extraGroupNames []string) []string { + groupNames := slices.Clone(env.Config().GroupsByComponent[componentName]) + + for _, groupName := range extraGroupNames { + if !slices.Contains(groupNames, groupName) { + groupNames = append(groupNames, groupName) + } + } + + return groupNames +} + // validateLockFiles checks lock file consistency against the resolved component // set. Skipped when skipValidation is true (set per-filter or via the global // '--skip-lock-validation' flag). diff --git a/internal/app/azldev/core/components/resolver_test.go b/internal/app/azldev/core/components/resolver_test.go index dd734667..39a49a4f 100644 --- a/internal/app/azldev/core/components/resolver_test.go +++ b/internal/app/azldev/core/components/resolver_test.go @@ -201,9 +201,9 @@ func TestFindComponents_FoundGroups(t *testing.T) { } // Setup the specs and compose a list of expected components. - expectedComponentConfigs := []projectconfig.ComponentConfig{} - for _, specPath := range specPaths { - expectedComponentConfigs = append(expectedComponentConfigs, setupTestSpec(t, env, specPath)) + expectedComponentConfigs := make([]projectconfig.ComponentConfig, len(specPaths)) + for idx, specPath := range specPaths { + expectedComponentConfigs[idx] = setupTestSpec(t, env, specPath) } // Find! @@ -705,6 +705,92 @@ func TestFindAllSpecPaths_MultipleSpecs(t *testing.T) { assert.ElementsMatch(t, []string{"/specs/a/test-a.spec", "/specs/b/test-b.spec"}, specPaths) } +func TestFindComponents_SpecDiscoveredComponentInheritsOverlayFilesFromDefaults(t *testing.T) { + env := testutils.NewTestEnv(t) + + touchFile(t, env, "/specs/ant/ant.spec") + require.NoError(t, fileutils.WriteFile(env.TestFS, + "/specs/ant/overlays/0001-branding.overlay.toml", + []byte(` +[metadata] +category = "azl-branding-policy" + +[[overlays]] +type = "spec-set-tag" +tag = "Vendor" +value = "Microsoft" +`), fileperms.PrivateFile)) + + env.Config.ComponentGroups["specs"] = projectconfig.ComponentGroupConfig{ + SpecPathPatterns: []string{"/specs/**/*.spec"}, + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, + }, + } + + filter := &components.ComponentFilter{ + IncludeAllComponents: true, + SkipLockValidation: true, + } + + resolved, err := components.NewResolver(env.Env).FindComponents(filter) + require.NoError(t, err) + + comp, ok := resolved.TryGet("ant") + require.True(t, ok) + + overlays := comp.GetConfig().Overlays + require.Len(t, overlays, 1) + assert.Empty(t, comp.GetConfig().OverlayFiles) + assert.Equal(t, projectconfig.ComponentOverlaySetSpecTag, overlays[0].Type) + assert.Equal(t, "Vendor", overlays[0].Tag) + require.NotNil(t, overlays[0].Metadata) + assert.Equal(t, projectconfig.OverlayCategoryAZLBrandingPolicy, overlays[0].Metadata.Category) +} + +func TestFindComponents_SpecPathFilterInheritsOverlayFilesFromGroupDefaults(t *testing.T) { + env := testutils.NewTestEnv(t) + + touchFile(t, env, "/specs/ant/ant.spec") + require.NoError(t, fileutils.WriteFile(env.TestFS, + "/specs/ant/overlays/0001-branding.overlay.toml", + []byte(` +[metadata] +category = "azl-branding-policy" + +[[overlays]] +type = "spec-set-tag" +tag = "Vendor" +value = "Microsoft" +`), fileperms.PrivateFile)) + + env.Config.ComponentGroups["specs"] = projectconfig.ComponentGroupConfig{ + SpecPathPatterns: []string{"/specs/**/*.spec"}, + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, + }, + } + + filter := &components.ComponentFilter{ + SpecPaths: []string{"/specs/ant/ant.spec"}, + SkipLockValidation: true, + } + + resolved, err := components.NewResolver(env.Env).FindComponents(filter) + require.NoError(t, err) + + comp, ok := resolved.TryGet("ant") + require.True(t, ok) + + overlays := comp.GetConfig().Overlays + require.Len(t, overlays, 1) + assert.Empty(t, comp.GetConfig().OverlayFiles) + assert.Equal(t, projectconfig.ComponentOverlaySetSpecTag, overlays[0].Type) + assert.Equal(t, "Vendor", overlays[0].Tag) + require.NotNil(t, overlays[0].Metadata) + assert.Equal(t, projectconfig.OverlayCategoryAZLBrandingPolicy, overlays[0].Metadata.Category) +} + // When a lock file exists for a component, the resolver should attach all of // its data (commit, import-commit, manual-bump, fingerprint) to the resolved // component via the Locked field — without touching the original Spec config. diff --git a/internal/projectconfig/component.go b/internal/projectconfig/component.go index b481b35b..dce2f264 100644 --- a/internal/projectconfig/component.go +++ b/internal/projectconfig/component.go @@ -273,8 +273,8 @@ type ComponentConfig struct { // Overlays to apply to sources after they've been acquired. May mutate the spec as well as sources. Overlays []ComponentOverlay `toml:"overlays,omitempty" json:"overlays,omitempty" table:"-" jsonschema:"title=Overlays,description=Overlays to apply to this component's spec and/or sources"` - // OverlayFiles, if set, lists glob patterns (relative to this component config file) - // matched against the filesystem at load time to locate per-file overlay documents. + // OverlayFiles, if set, lists path or glob patterns (relative to this component config file) + // matched against the filesystem after component config resolution to locate per-file overlay documents. // Each matched file is parsed as an [OverlayFile]: one logical change consisting of a // file-level `[metadata]` block plus an ordered list of `[[overlays]]`. The per-file // metadata is stamped onto every overlay in the file. Matches are concatenated in the @@ -282,9 +282,11 @@ type ComponentConfig struct { // filename (lexicographic) order, using the full path as a tie-breaker when // filenames match. Duplicate matches are de-duplicated, preserving first // occurrence. The resulting overlays are appended to [ComponentConfig.Overlays] - // after any inline overlays. Excluded from the fingerprint because the value affects - // only where overlays are sourced from, not their content. - OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Glob patterns (relative to the component config file) matched against the filesystem to locate per-file overlay documents at load time" fingerprint:"-"` + // after any inline overlays. A value set in a higher-priority config layer replaces + // lower-priority overlay-files values; an explicit empty list disables inherited + // overlay files. Excluded from the fingerprint because the value affects only where + // overlays are sourced from, not their content. + OverlayFiles []string `toml:"overlay-files,omitempty" json:"overlayFiles,omitempty" table:"-" validate:"dive,required" jsonschema:"title=Overlay files,description=Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" fingerprint:"-"` // Configuration for building the component. Build ComponentBuildConfig `toml:"build,omitempty" json:"build,omitempty" table:"-" jsonschema:"title=Build configuration,description=Configuration for building the component"` @@ -317,11 +319,17 @@ var AllowedSourceFilesHashTypes = map[fileutils.HashType]bool{ // Mutates the component config, updating it with overrides present in other. func (c *ComponentConfig) MergeUpdatesFrom(other *ComponentConfig) error { + otherOverlayFiles := slices.Clone(other.OverlayFiles) + err := mergo.Merge(c, other, mergo.WithOverride, mergo.WithAppendSlice) if err != nil { return fmt.Errorf("failed to merge project info:\n%w", err) } + if other.OverlayFiles != nil { + c.OverlayFiles = otherOverlayFiles + } + return nil } @@ -396,9 +404,9 @@ func (c *ComponentConfig) WithAbsolutePaths(referenceDir string) *ComponentConfi SourceFiles: deep.MustCopy(c.SourceFiles), Packages: deep.MustCopy(c.Packages), Publish: deep.MustCopy(c.Publish), - // OverlayFiles is consumed at load time (see applyOverlayFiles) before paths are - // absolutized; nothing reads it afterward, so preserve the original value verbatim - // rather than redundantly re-rooting each glob. + // OverlayFiles is consumed after component config resolution; preserve it verbatim + // here so inherited patterns can be interpreted relative to the concrete component + // config file. OverlayFiles: slices.Clone(c.OverlayFiles), } diff --git a/internal/projectconfig/component_test.go b/internal/projectconfig/component_test.go index 2acc9a7e..35dd7d74 100644 --- a/internal/projectconfig/component_test.go +++ b/internal/projectconfig/component_test.go @@ -183,6 +183,45 @@ func TestMergeComponentUpdates(t *testing.T) { require.Equal(t, []string{"x", "y", "w"}, base.Build.Without) } +func TestMergeComponentUpdates_OverlayFilesOverride(t *testing.T) { + base := projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, + } + + updates := projectconfig.ComponentConfig{ + OverlayFiles: []string{"custom/*.overlay.toml"}, + } + + err := base.MergeUpdatesFrom(&updates) + require.NoError(t, err) + require.Equal(t, []string{"custom/*.overlay.toml"}, base.OverlayFiles) +} + +func TestMergeComponentUpdates_OverlayFilesEmptyOverride(t *testing.T) { + base := projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, + } + + updates := projectconfig.ComponentConfig{ + OverlayFiles: []string{}, + } + + err := base.MergeUpdatesFrom(&updates) + require.NoError(t, err) + require.NotNil(t, base.OverlayFiles) + require.Empty(t, base.OverlayFiles) +} + +func TestMergeComponentUpdates_OverlayFilesInheritWhenUnset(t *testing.T) { + base := projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, + } + + err := base.MergeUpdatesFrom(&projectconfig.ComponentConfig{}) + require.NoError(t, err) + require.Equal(t, []string{"overlays/*.overlay.toml"}, base.OverlayFiles) +} + func TestAllowedSourceFilesHashTypes_MatchesJSONSchemaEnum(t *testing.T) { // Extract enum values from the jsonschema tag on // [projectconfig.SourceFileReference.HashType]. @@ -325,6 +364,27 @@ func TestResolveComponentConfig(t *testing.T) { assert.Equal(t, "comp-commit", resolved.Spec.UpstreamCommit) }) + t.Run("component empty overlay files clears group defaults", func(t *testing.T) { + groups := map[string]projectconfig.ComponentGroupConfig{ + "core": { + DefaultComponentConfig: projectconfig.ComponentConfig{ + OverlayFiles: []string{"overlays/*.overlay.toml"}, + }, + }, + } + comp := projectconfig.ComponentConfig{ + Name: "curl", + OverlayFiles: []string{}, + } + + resolved, err := projectconfig.ResolveComponentConfig( + comp, projectconfig.ComponentConfig{}, distroDefaults, groups, []string{"core"}, + ) + require.NoError(t, err) + require.NotNil(t, resolved.OverlayFiles) + assert.Empty(t, resolved.OverlayFiles) + }) + t.Run("missing group errors", func(t *testing.T) { comp := projectconfig.ComponentConfig{Name: "curl"} diff --git a/internal/projectconfig/loader.go b/internal/projectconfig/loader.go index ab6227b5..31971f90 100644 --- a/internal/projectconfig/loader.go +++ b/internal/projectconfig/loader.go @@ -433,13 +433,6 @@ func loadProjectConfigFile( cfg.sourcePath = absFilePath cfg.dir = filepath.Dir(absFilePath) - // Resolve per-component overlay file globs, stamping each - // file's [metadata] onto its overlays and appending them to the component before - // validation runs. - if err := applyOverlayFiles(fs, cfg, permissiveConfigParsing); err != nil { - return nil, err - } - // Make sure that the read data is internally consistent. err = cfg.Validate() if err != nil { diff --git a/internal/projectconfig/overlay_file.go b/internal/projectconfig/overlay_file.go index eecb2d22..eb04ff25 100644 --- a/internal/projectconfig/overlay_file.go +++ b/internal/projectconfig/overlay_file.go @@ -31,22 +31,12 @@ var ErrOverlayFilePerOverlayMetadata = errors.New( // and would otherwise silently contribute nothing. var ErrOverlayFileEmpty = errors.New("overlay file declares no overlays") -// ErrOverlayFilesNoMatches is returned when one of a component's -// [ComponentConfig.OverlayFiles] glob patterns matches no files on disk. A configured -// glob that matches nothing is almost always a misconfiguration (wrong path or -// missing files) and is surfaced as an error rather than silently contributing -// nothing. +// ErrOverlayFilesNoMatches is returned when a literal [ComponentConfig.OverlayFiles] +// path matches no files on disk. Glob patterns may intentionally match no files when +// inherited across many components, but a literal path with no match is almost always +// a typo. var ErrOverlayFilesNoMatches = errors.New("overlay-files pattern matched no files") -// ErrOverlayFilesInDefaultConfig is returned when `overlay-files` is set on a -// default component config. Overlay file globs are resolved per concrete component -// before default configs are merged in, so a value set on a default would be -// silently ignored. Until overlay-files is wired through the default-merge path, -// declaring it on a default config is rejected. -var ErrOverlayFilesInDefaultConfig = errors.New( - "overlay-files is not supported on default-component-config; set it on individual components", -) - // OverlayFile is the on-disk representation of a single overlay document. Each file // represents one logical change: the file-level [OverlayFile.Metadata] is applied // to every overlay in [OverlayFile.Overlays] at load time, after which the resulting @@ -61,62 +51,45 @@ type OverlayFile struct { Overlays []ComponentOverlay `toml:"overlays"` } -// applyOverlayFiles resolves each component's [ComponentConfig.OverlayFiles] glob -// patterns (when set), parses every matched file as an [OverlayFile] in deterministic -// order, stamps the per-file metadata onto each overlay, and appends the resulting -// overlays to the component's [ComponentConfig.Overlays] slice. Inline overlays -// declared directly on the component come first; file-sourced overlays are appended -// after them. -// -// Called from [loadProjectConfigFile] after TOML decode but before [ConfigFile.Validate], -// so all per-overlay validation rules apply uniformly regardless of declaration site. -func applyOverlayFiles(fs opctx.FS, cfg *ConfigFile, permissiveConfigParsing bool) error { - if err := rejectOverlayFilesInDefaults(cfg); err != nil { - return err +// ExpandResolvedOverlayFiles resolves a component's post-resolution +// [ComponentConfig.OverlayFiles] glob patterns, parses every matched file as an +// [OverlayFile], stamps the per-file metadata onto each overlay, appends the +// resulting overlays after inline overlays, and clears [ComponentConfig.OverlayFiles] +// so the returned config is not expanded twice. +func ExpandResolvedOverlayFiles( + fs opctx.FS, component ComponentConfig, referenceDir string, permissiveConfigParsing bool, +) (ComponentConfig, error) { + if len(component.OverlayFiles) == 0 { + return component, nil } - for componentName, component := range cfg.Components { - if len(component.OverlayFiles) == 0 { - continue - } - - loaded, err := loadOverlayFiles(fs, cfg.dir, component.OverlayFiles, permissiveConfigParsing) - if err != nil { - return fmt.Errorf("component %#q overlay-files:\n%w", componentName, err) - } - - component.Overlays = append(component.Overlays, loaded...) - cfg.Components[componentName] = component + if referenceDir == "" && overlayFilesHasRelativePattern(component.OverlayFiles) { + return ComponentConfig{}, fmt.Errorf( + "component %#q has relative 'overlay-files' entries but no reference directory", + component.Name, + ) } - return nil -} - -// rejectOverlayFilesInDefaults returns [ErrOverlayFilesInDefaultConfig] if any default -// component config in cfg sets `overlay-files`. Overlay file globs are resolved per -// concrete component (see [applyOverlayFiles]) before default configs are merged, so a -// value declared on a default would be silently dropped. This stopgap surfaces the -// misconfiguration until overlay-files is plumbed through the default-merge path. -func rejectOverlayFilesInDefaults(cfg *ConfigFile) error { - if cfg.DefaultComponentConfig != nil && len(cfg.DefaultComponentConfig.OverlayFiles) > 0 { - return fmt.Errorf("%w (project-level default-component-config)", ErrOverlayFilesInDefaultConfig) + loaded, err := loadOverlayFiles(fs, referenceDir, component.OverlayFiles, permissiveConfigParsing) + if err != nil { + return ComponentConfig{}, fmt.Errorf("component %#q overlay-files:\n%w", component.Name, err) } - for name, group := range cfg.ComponentGroups { - if len(group.DefaultComponentConfig.OverlayFiles) > 0 { - return fmt.Errorf("%w (component-group %#q)", ErrOverlayFilesInDefaultConfig, name) - } - } + component.Overlays = append(component.Overlays, loaded...) + // Clear the glob list after expansion so later resolver paths do not append the same overlays again. + component.OverlayFiles = nil - for name, distro := range cfg.Distros { - for version, versionDef := range distro.Versions { - if len(versionDef.DefaultComponentConfig.OverlayFiles) > 0 { - return fmt.Errorf("%w (distro %#q version %#q)", ErrOverlayFilesInDefaultConfig, name, version) - } + return component, nil +} + +func overlayFilesHasRelativePattern(patterns []string) bool { + for _, pattern := range patterns { + if !filepath.IsAbs(pattern) { + return true } } - return nil + return false } // loadOverlayFiles resolves each glob pattern (relative to referenceDir if not @@ -127,8 +100,8 @@ func rejectOverlayFilesInDefaults(cfg *ConfigFile) error { // Matches are concatenated in the order patterns are declared; within a single // pattern, matches are applied in filename (lexicographic) order, using the full // path as a tie-breaker when filenames match. Duplicate matches across patterns -// are de-duplicated, preserving first occurrence. Each pattern is required to -// match at least one file. +// are de-duplicated, preserving first occurrence. Glob patterns that match no +// files contribute no overlays; literal paths must match a file. func loadOverlayFiles( fs opctx.FS, referenceDir string, patterns []string, permissiveConfigParsing bool, ) ([]ComponentOverlay, error) { @@ -146,17 +119,16 @@ func loadOverlayFiles( matches, err := fileutils.Glob( fs, absPattern, doublestar.WithFailOnIOErrors(), - doublestar.WithFailOnPatternNotExist(), doublestar.WithFilesOnly(), ) switch { - case errors.Is(err, doublestar.ErrPatternNotExist): - return nil, fmt.Errorf("%w: %q", ErrOverlayFilesNoMatches, pattern) case err != nil: return nil, fmt.Errorf("failed to scan for overlay files matching %q:\n%w", pattern, err) - case len(matches) == 0: + case len(matches) == 0 && !containsPattern(pattern): return nil, fmt.Errorf("%w: %q", ErrOverlayFilesNoMatches, pattern) + case len(matches) == 0: + continue } slices.SortFunc(matches, func(left, right string) int { @@ -246,7 +218,11 @@ func loadOverlayFile( } overlay.Metadata = stampedMetadata.clone() + overlay.Source = makeAbsolute(overlayDir, overlay.Source) + if err := overlay.Validate(); err != nil { + return nil, fmt.Errorf("invalid overlay %d in overlay file %q:\n%w", idx+1, overlayPath, err) + } } return ofile.Overlays, nil diff --git a/internal/projectconfig/overlay_file_test.go b/internal/projectconfig/overlay_file_test.go index d714dc00..408ac4ce 100644 --- a/internal/projectconfig/overlay_file_test.go +++ b/internal/projectconfig/overlay_file_test.go @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. -//nolint:testpackage // Allow to test private functions (i.e., applyOverlayFiles). +//nolint:testpackage // Allow to test private functions (i.e., loadOverlayFiles). package projectconfig import ( @@ -149,30 +149,58 @@ value = "fourth" assert.Equal(t, "fourth", loaded[3].Value) } -func TestLoadOverlayFiles_NoMatchesIsError(t *testing.T) { +func TestLoadOverlayFiles_NoMatchesIsNoop(t *testing.T) { ctx := testctx.NewCtx() overlayDir := "/project/comps/empty/overlays" // A non-overlay file alongside must not be picked up; a glob that matches no - // files is a misconfiguration and must error. + // files contributes no overlays. require.NoError(t, fileutils.WriteFile(ctx.FS(), filepath.Join(overlayDir, "README.md"), []byte("not an overlay"), fileperms.PrivateFile)) - _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) - require.ErrorIs(t, err, ErrOverlayFilesNoMatches) + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + require.NoError(t, err) + assert.Empty(t, loaded) } -func TestLoadOverlayFiles_MissingDirIsError(t *testing.T) { +func TestLoadOverlayFiles_MissingDirIsNoop(t *testing.T) { ctx := testctx.NewCtx() - _, err := loadOverlayFiles( + loaded, err := loadOverlayFiles( ctx.FS(), "/project", []string{overlayGlob("/project/comps/does-not-exist/overlays")}, false, ) + require.NoError(t, err) + assert.Empty(t, loaded) +} + +func TestLoadOverlayFiles_MissingLiteralPathErrors(t *testing.T) { + ctx := testctx.NewCtx() + + _, err := loadOverlayFiles( + ctx.FS(), "/project", + []string{"/project/comps/does-not-exist/overlays/0001-missing.overlay.toml"}, false, + ) require.ErrorIs(t, err, ErrOverlayFilesNoMatches) } +func TestLoadOverlayFiles_NoMatchDoesNotBlockLaterMatches(t *testing.T) { + ctx := testctx.NewCtx() + overlayDir := antOverlayDir + + require.NoError(t, fileutils.WriteFile(ctx.FS(), + filepath.Join(overlayDir, "0001-backport.overlay.toml"), + []byte(validBackportOverlayFile), fileperms.PrivateFile)) + + loaded, err := loadOverlayFiles(ctx.FS(), "/project", []string{ + overlayGlob("/project/comps/does-not-exist/overlays"), + overlayGlob(overlayDir), + }, false) + require.NoError(t, err) + require.Len(t, loaded, 2) +} + func TestLoadOverlayFiles_RejectsPerOverlayMetadata(t *testing.T) { ctx := testctx.NewCtx() overlayDir := badOverlayTestDir @@ -231,6 +259,28 @@ value = "Microsoft" assert.Contains(t, err.Error(), "category") } +func TestLoadOverlayFiles_RejectsInvalidOverlay(t *testing.T) { + ctx := testctx.NewCtx() + overlayDir := badOverlayTestDir + + require.NoError(t, fileutils.WriteFile(ctx.FS(), + filepath.Join(overlayDir, "0001-bad.overlay.toml"), + []byte(` +[metadata] +category = "azl-branding-policy" + +[[overlays]] +type = "spec-set-tag" +tag = "Vendor" +`), fileperms.PrivateFile)) + + _, err := loadOverlayFiles(ctx.FS(), "/project", []string{overlayGlob(overlayDir)}, false) + require.Error(t, err) + assert.Contains(t, err.Error(), "invalid overlay 1") + assert.Contains(t, err.Error(), "requires") + assert.Contains(t, err.Error(), "value") +} + func TestLoadOverlayFile_PermissiveTolerates_InvalidMetadata(t *testing.T) { ctx := testctx.NewCtx() overlayPath := filepath.Join(badOverlayTestDir, "0001-bad.overlay.toml") @@ -319,7 +369,7 @@ func TestLoadOverlayFiles_DoubleStarMatchesNested(t *testing.T) { require.Len(t, loaded, 2, "**-style glob must descend into subdirectories") } -func TestApplyOverlayFiles_AppendsAfterInlineOverlays(t *testing.T) { +func TestExpandResolvedOverlayFiles_AppendsAfterInlineOverlays(t *testing.T) { ctx := testctx.NewCtx() overlayDir := antOverlayDir @@ -327,147 +377,60 @@ func TestApplyOverlayFiles_AppendsAfterInlineOverlays(t *testing.T) { filepath.Join(overlayDir, "0001-backport.overlay.toml"), []byte(validBackportOverlayFile), fileperms.PrivateFile)) - cfg := &ConfigFile{ - dir: "/project", - Components: map[string]ComponentConfig{ - "ant": { - OverlayFiles: []string{"comps/ant/overlays/*.overlay.toml"}, - Overlays: []ComponentOverlay{ - { - Type: ComponentOverlaySetSpecTag, - Tag: "Vendor", - Value: "Microsoft", - Metadata: &OverlayMetadata{ - Category: OverlayCategoryAZLBrandingPolicy, - }, - }, + cfg := &ConfigFile{dir: "/project"} + component := ComponentConfig{ + Name: "ant", + SourceConfigFile: cfg, + OverlayFiles: []string{"comps/ant/overlays/*.overlay.toml"}, + Overlays: []ComponentOverlay{ + { + Type: ComponentOverlaySetSpecTag, + Tag: "Vendor", + Value: "Microsoft", + Metadata: &OverlayMetadata{ + Category: OverlayCategoryAZLBrandingPolicy, }, }, }, } - require.NoError(t, applyOverlayFiles(ctx.FS(), cfg, false)) + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), component, "/project", false) + require.NoError(t, err) - ant := cfg.Components["ant"] - require.Len(t, ant.Overlays, 3, "inline overlay + two file-sourced overlays") + require.Len(t, expanded.Overlays, 3, "inline overlay + two file-sourced overlays") + assert.Empty(t, expanded.OverlayFiles, "overlay-files should be consumed after expansion") // Inline first. - assert.Equal(t, ComponentOverlaySetSpecTag, ant.Overlays[0].Type) - assert.Equal(t, OverlayCategoryAZLBrandingPolicy, ant.Overlays[0].Metadata.Category) + assert.Equal(t, ComponentOverlaySetSpecTag, expanded.Overlays[0].Type) + assert.Equal(t, OverlayCategoryAZLBrandingPolicy, expanded.Overlays[0].Metadata.Category) // File-sourced overlays appended in file/declaration order, with the file's // metadata stamped onto each. - assert.Equal(t, ComponentOverlaySearchAndReplaceInSpec, ant.Overlays[1].Type) - require.NotNil(t, ant.Overlays[1].Metadata) - assert.Equal(t, OverlayCategoryBackportDistGit, ant.Overlays[1].Metadata.Category) + assert.Equal(t, ComponentOverlaySearchAndReplaceInSpec, expanded.Overlays[1].Type) + require.NotNil(t, expanded.Overlays[1].Metadata) + assert.Equal(t, OverlayCategoryBackportDistGit, expanded.Overlays[1].Metadata.Category) - assert.Equal(t, ComponentOverlayRemoveSubpackage, ant.Overlays[2].Type) - require.NotNil(t, ant.Overlays[2].Metadata) - assert.Equal(t, OverlayCategoryBackportDistGit, ant.Overlays[2].Metadata.Category) + assert.Equal(t, ComponentOverlayRemoveSubpackage, expanded.Overlays[2].Type) + require.NotNil(t, expanded.Overlays[2].Metadata) + assert.Equal(t, OverlayCategoryBackportDistGit, expanded.Overlays[2].Metadata.Category) } -func TestApplyOverlayFiles_NoopWhenUnset(t *testing.T) { +func TestExpandResolvedOverlayFiles_NoopWhenUnset(t *testing.T) { ctx := testctx.NewCtx() - cfg := &ConfigFile{ - dir: "/project", - Components: map[string]ComponentConfig{ - "ant": { - Overlays: []ComponentOverlay{ - {Type: ComponentOverlayAddSpecTag, Tag: "Vendor", Value: "Microsoft"}, - }, - }, - }, - } - - require.NoError(t, applyOverlayFiles(ctx.FS(), cfg, false)) - - ant := cfg.Components["ant"] - require.Len(t, ant.Overlays, 1, "no overlay-files, no merging") -} - -func TestRejectOverlayFilesInDefaults(t *testing.T) { - const overlayFilePattern = "overlays/*.overlay.toml" - - testCases := []struct { - name string - cfg ConfigFile - errorContains string - }{ - { - name: "project-level default-component-config", - cfg: ConfigFile{ - DefaultComponentConfig: &ComponentConfig{ - OverlayFiles: []string{overlayFilePattern}, - }, - }, - errorContains: "project-level default-component-config", - }, - { - name: "component-group default-component-config", - cfg: ConfigFile{ - ComponentGroups: map[string]ComponentGroupConfig{ - "core": { - DefaultComponentConfig: ComponentConfig{ - OverlayFiles: []string{overlayFilePattern}, - }, - }, - }, - }, - errorContains: "component-group `core`", - }, - { - name: "distro version default-component-config", - cfg: ConfigFile{ - Distros: map[string]DistroDefinition{ - "azl": { - Versions: map[string]DistroVersionDefinition{ - "3.0": { - DefaultComponentConfig: ComponentConfig{ - OverlayFiles: []string{overlayFilePattern}, - }, - }, - }, - }, - }, - }, - errorContains: "distro `azl` version `3.0`", - }, - { - name: "unset defaults", - cfg: ConfigFile{ - DefaultComponentConfig: &ComponentConfig{}, - ComponentGroups: map[string]ComponentGroupConfig{ - "core": {}, - }, - Distros: map[string]DistroDefinition{ - "azl": { - Versions: map[string]DistroVersionDefinition{ - "3.0": {}, - }, - }, - }, - }, + component := ComponentConfig{ + Name: "ant", + Overlays: []ComponentOverlay{ + {Type: ComponentOverlayAddSpecTag, Tag: "Vendor", Value: "Microsoft"}, }, } - for _, testCase := range testCases { - t.Run(testCase.name, func(t *testing.T) { - err := rejectOverlayFilesInDefaults(&testCase.cfg) - - if testCase.errorContains == "" { - require.NoError(t, err) - - return - } - - require.ErrorIs(t, err, ErrOverlayFilesInDefaultConfig) - assert.Contains(t, err.Error(), testCase.errorContains) - }) - } + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), component, "", false) + require.NoError(t, err) + require.Len(t, expanded.Overlays, 1, "no overlay-files, no merging") } -func TestApplyOverlayFiles_AcceptsAbsoluteGlob(t *testing.T) { +func TestExpandResolvedOverlayFiles_AcceptsAbsoluteGlob(t *testing.T) { ctx := testctx.NewCtx() absDir := "/elsewhere/overlays" @@ -475,43 +438,48 @@ func TestApplyOverlayFiles_AcceptsAbsoluteGlob(t *testing.T) { filepath.Join(absDir, "0001-backport.overlay.toml"), []byte(validBackportOverlayFile), fileperms.PrivateFile)) - cfg := &ConfigFile{ - dir: "/project", - Components: map[string]ComponentConfig{ - "ant": {OverlayFiles: []string{overlayGlob(absDir)}}, - }, - } + component := ComponentConfig{Name: "ant", OverlayFiles: []string{overlayGlob(absDir)}} - require.NoError(t, applyOverlayFiles(ctx.FS(), cfg, false)) + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), component, "", false) + require.NoError(t, err) - ant := cfg.Components["ant"] - require.Len(t, ant.Overlays, 2, "absolute glob is not re-rooted under cfg.dir") - require.NotNil(t, ant.Overlays[0].Metadata) - assert.Equal(t, OverlayCategoryBackportDistGit, ant.Overlays[0].Metadata.Category) + require.Len(t, expanded.Overlays, 2, "absolute glob is not re-rooted under cfg.dir") + require.NotNil(t, expanded.Overlays[0].Metadata) + assert.Equal(t, OverlayCategoryBackportDistGit, expanded.Overlays[0].Metadata.Category) } -// TestLoadAndResolveProjectConfig_OverlayFiles exercises the full loader pipeline -// (loadAndResolveProjectConfig -> loadProjectConfigFile -> applyOverlayFiles) and -// guards against regressions that drop the overlay-files hook from the loader. -func TestLoadAndResolveProjectConfig_OverlayFiles(t *testing.T) { +func TestExpandResolvedOverlayFiles_RequiresReferenceDirForRelativePattern(t *testing.T) { ctx := testctx.NewCtx() - require.NoError(t, fileutils.WriteFile(ctx.FS(), testConfigPath, []byte(` -[components.ant] -overlay-files = ["comps/ant/overlays/*.overlay.toml"] -`), fileperms.PrivateFile)) + _, err := ExpandResolvedOverlayFiles(ctx.FS(), ComponentConfig{ + Name: "ant", + OverlayFiles: []string{"overlays/*.overlay.toml"}, + }, "", false) + require.Error(t, err) + assert.Contains(t, err.Error(), "no reference directory") +} + +func TestExpandResolvedOverlayFiles_DefaultPatternUsesConcreteComponentConfigDir(t *testing.T) { + ctx := testctx.NewCtx() + componentConfig := &ConfigFile{dir: "/project/comps/ant"} require.NoError(t, fileutils.WriteFile(ctx.FS(), "/project/comps/ant/overlays/0001-backport.overlay.toml", []byte(validBackportOverlayFile), fileperms.PrivateFile)) - cfg, err := loadAndResolveProjectConfig(ctx.FS(), false, testConfigPath) + resolved, err := ResolveComponentConfig( + ComponentConfig{Name: "ant", SourceConfigFile: componentConfig}, + ComponentConfig{OverlayFiles: []string{"overlays/*.overlay.toml"}}, + ComponentConfig{}, + nil, + nil, + ) require.NoError(t, err) - require.NotNil(t, cfg) - ant, ok := cfg.Components["ant"] - require.True(t, ok, "ant component should be present") - require.Len(t, ant.Overlays, 2) - require.NotNil(t, ant.Overlays[0].Metadata) - assert.Equal(t, OverlayCategoryBackportDistGit, ant.Overlays[0].Metadata.Category) + expanded, err := ExpandResolvedOverlayFiles(ctx.FS(), resolved, componentConfig.Dir(), false) + require.NoError(t, err) + require.Len(t, expanded.Overlays, 2) + assert.Empty(t, expanded.OverlayFiles) + require.NotNil(t, expanded.Overlays[0].Metadata) + assert.Equal(t, OverlayCategoryBackportDistGit, expanded.Overlays[0].Metadata.Category) } diff --git a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap index 7b824f4e..d665b45f 100755 --- a/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshotsContainer_config_generate-schema_stdout_1.snap @@ -141,7 +141,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Glob patterns (relative to the component config file) matched against the filesystem to locate per-file overlay documents at load time" + "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap index 7b824f4e..d665b45f 100755 --- a/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap +++ b/scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap @@ -141,7 +141,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Glob patterns (relative to the component config file) matched against the filesystem to locate per-file overlay documents at load time" + "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" }, "build": { "$ref": "#/$defs/ComponentBuildConfig", diff --git a/schemas/azldev.schema.json b/schemas/azldev.schema.json index 7b824f4e..d665b45f 100644 --- a/schemas/azldev.schema.json +++ b/schemas/azldev.schema.json @@ -141,7 +141,7 @@ }, "type": "array", "title": "Overlay files", - "description": "Glob patterns (relative to the component config file) matched against the filesystem to locate per-file overlay documents at load time" + "description": "Path or glob patterns (relative to the component config file or matched spec directory) matched against the filesystem to locate per-file overlay documents after component config resolution. Use an empty list to disable inherited overlay-file patterns" }, "build": { "$ref": "#/$defs/ComponentBuildConfig",