|
| 1 | +# New resolver and download configuration |
| 2 | + |
| 3 | +- Author: Christian Heimes |
| 4 | +- Created: 2026-02-24 |
| 5 | +- Status: Open |
| 6 | + |
| 7 | +## What |
| 8 | + |
| 9 | +This enhancement document proposes a new approach to configure the package |
| 10 | +resolver and source / sdist downloader. The new settings are covering a |
| 11 | +wider range of use cases. Common patterns like building a package from a |
| 12 | +git checkout will no longer need custom Python plugins. |
| 13 | + |
| 14 | +## Why |
| 15 | + |
| 16 | +In downstream, we are encountering an increasing amount of packages that do |
| 17 | +not build from sdists on PyPI. Either package maintainers are not uploading |
| 18 | +source distributions to PyPI or sdists have issues. In some cases, packages |
| 19 | +use a midstream fork that is not on PyPI. The sources need to be build from |
| 20 | +git. |
| 21 | + |
| 22 | +Because Fromager \<= 0.76 does not have declarative settings for GitHub/GitLab |
| 23 | +resolver or cloning git repositories, we have to write custom Python plugins. |
| 24 | +The plugins are a maintenance burden. |
| 25 | + |
| 26 | +## Goals |
| 27 | + |
| 28 | +- support common use cases with package settings instead of custom plugin code |
| 29 | +- cover most common resolver scenarios: |
| 30 | + - resolve package on PyPI (sdist, wheel, or both) |
| 31 | + - resolve package on GitHub or GitLab with custom tag matcher |
| 32 | +- cover common sdist download and build scenarios: |
| 33 | + - sdist from PyPI |
| 34 | + - prebuilt wheel from PyPI |
| 35 | + - download tarball from URL |
| 36 | + - clone git repository |
| 37 | + - download an artifact from GitHub / GitLab release or tag |
| 38 | + - build sdist with PEP 517 hook or plain tarball |
| 39 | +- support per-variant setting, e.g. one variant uses prebuilt wheel while the |
| 40 | + rest uses sdist. |
| 41 | +- gradual migration path from old system to new configuration |
| 42 | + |
| 43 | +## Non-goals |
| 44 | + |
| 45 | +- The new system will not cover all use cases. Some specific use cases will |
| 46 | + still require custom code. |
| 47 | +- Retrieval of additional sources is out of scope, e.g. a package `egg` that |
| 48 | + needs `libegg-{version}.tar.gz`. |
| 49 | +- Provide SSH transport for git. The feature can be added at a later point |
| 50 | + when it's needed. |
| 51 | +- Extra options for authentication. The `requests` library and `git` CLI can |
| 52 | + use `$HOME/.netrc` for authentication. |
| 53 | + > **NOTE:** `requests` also supports `NETRC` environment variable, |
| 54 | + > `libcurl` and therefore `git` did not support `NETRC` before |
| 55 | + > libcurl [8.16.0](https://curl.se/ch/8.16.0.html) (2025-09-10). Before |
| 56 | + > `git` _only_ supports `$HOME/.netrc`. |
| 57 | +
|
| 58 | +## How |
| 59 | + |
| 60 | +The new system will use a new top-level configuration key `source`. The old |
| 61 | +`download_source` and `resolver_dist` settings will stay supported for a |
| 62 | +while. Eventually the old options will be deprecated and removed. |
| 63 | + |
| 64 | +In addition to a top-level `source` entry, the resolver and downloader can |
| 65 | +be overwritten in a variant `source` entry. This enables variant-specific |
| 66 | +settings. For example a package resolve and download from GitHub and one |
| 67 | +variant uses a pre-built wheel from a custom index. |
| 68 | + |
| 69 | +Each use case is handled a provider profile. The profile name acts as a tag |
| 70 | +([discriminated union](https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions)). |
| 71 | +Each use case has a well-defined set of mandatory and optional arguments. |
| 72 | + |
| 73 | +**Example:** |
| 74 | + |
| 75 | +```yaml |
| 76 | +source: |
| 77 | + # `pypi-sdist` is the default provider |
| 78 | + provider: pypi-sdist |
| 79 | +variants: |
| 80 | + egg: |
| 81 | + source: |
| 82 | + # resolve and download prebuilt wheel |
| 83 | + provider: pypi-prebuilt |
| 84 | + index_url: https://custom-index.example/simple |
| 85 | + spam: |
| 86 | + source: |
| 87 | + # resolve tag on GitLab, clone tag over https, build an sdist with PEP 517 hook |
| 88 | + provider: gitlab-tag-git |
| 89 | + project_url: https://gitlab.example/spam/spam |
| 90 | + matcher_factory: package_plugins.matchers:midstream_matcher_factory |
| 91 | + build_sdist: pep517 |
| 92 | + viking: |
| 93 | + source: |
| 94 | + # resolve on PyPI, git clone, and build as tarball |
| 95 | + provider: pypi-git |
| 96 | + clone_url: https://git.example/viking/viking.git |
| 97 | + tag: 'v{version}' |
| 98 | + build_sdist: tarball |
| 99 | + caerbannog: |
| 100 | + # resolve with a mapping of version number to git refs, git clone, and build with PEP 517 hook |
| 101 | + source: |
| 102 | + provider: versionmap-git |
| 103 | + clone_url: https://git.example/viking/viking.git |
| 104 | + build_sdist: pep517 |
| 105 | + versionmap: |
| 106 | + '1.0': abad1dea |
| 107 | + '1.1': refs/tags/1.1 |
| 108 | + camelot: |
| 109 | + source: |
| 110 | + # On second thought, let's not go to Camelot. It is a silly place. |
| 111 | + provider: not-available |
| 112 | +``` |
| 113 | +
|
| 114 | +### Profiles |
| 115 | +
|
| 116 | +- The `pypi-sdist` profile resolve versions on PyPI or PyPI-compatible index. |
| 117 | + It only takes sdists into account and downloads the sdist from the index. |
| 118 | + The profile is equivalent to the current default settings with |
| 119 | + `include_sdists: true` and `include_wheels: false`. |
| 120 | + |
| 121 | +- The `pypi-prebuilt` profile resolve versions of platform-specific wheels |
| 122 | + on PyPI and downloads the pre-built wheel. The profile is equivalent to |
| 123 | + `include_sdists: false`, `include_wheels: true`, and variant setting |
| 124 | + `pre_build: true`. |
| 125 | + |
| 126 | +- The `pypi-download` resolve versions of any package on PyPI and downloads |
| 127 | + a tarball from an external URL (with `{version}` variable in download URL). |
| 128 | + It takes any sdist and any wheel into account. The profile is equivalent |
| 129 | + with `include_sdists: true`, `include_wheels: true`, `ignore_platform: true`, |
| 130 | + and a `download_source.url`. |
| 131 | + |
| 132 | +- The `pypi-git` is similar to the `pypi-download` profile. Instead of |
| 133 | + downloading a tarball, it clones a git repository at a specific tag. |
| 134 | + |
| 135 | +- The `versionmap-git` profiles maps known version numbers to known git |
| 136 | + commits. It clones a git repo at the configured tag. |
| 137 | + |
| 138 | +- The `gitlab-tag-git` and `github-tag-git` profiles use the |
| 139 | + `GitLabTagProvider` or `GitHubTagProvider` to resolve versions. The |
| 140 | + profiles git clone a project over `https` or `ssh` protocol. |
| 141 | + |
| 142 | +- The `gitlab-tag-download` and `github-tag-download` are similar to |
| 143 | + `gitlab-tag-git` and `github-tag-git` profiles. Instead of cloning a git |
| 144 | + repository, they download a git tarball or an release artifact. |
| 145 | + |
| 146 | +- The `hooks` profile calls the `resolver_provider` and `download_source` |
| 147 | + [hooks](../reference/hooks.rst). |
| 148 | + |
| 149 | +- The `not-available` profile raises an error. It can be used to block a |
| 150 | + package and only enable it for a single variant. |
| 151 | + |
| 152 | +### default behavior and hooks |
| 153 | + |
| 154 | +When a package setting file does not have a top-level `source` configuration, |
| 155 | +then Fromager keep its old behavior. It first looks for `resolver_provider` |
| 156 | +and `download_source` [hooks](../reference/hooks.rst), then looks for source |
| 157 | +distributions on PyPI. |
| 158 | + |
| 159 | +When a package has a plugin with a `resolver_provider` or `download_source` |
| 160 | +hook and `source` settings, then at least one `source` setting (top-level or |
| 161 | +variant) must use `provider: hooks`. The rule ensures that the hooks are |
| 162 | +used. |
| 163 | + |
| 164 | +### git clone |
| 165 | + |
| 166 | +Like pip's VCS feature, all git clone operations automatically retrieve all |
| 167 | +submodules recursively. The final sdist does not include a `.git` directory. |
| 168 | +Instead Fromager generates a `.git_archival.txt` file for setuptools-scm's |
| 169 | +[builtin mechanism for obtaining version numbers](https://setuptools-scm.readthedocs.io/en/latest/usage/#builtin-mechanisms-for-obtaining-version-numbers). |
| 170 | + |
| 171 | +The resolver and `Candidate` class do not support VCS URLs, yet. Fromager can |
| 172 | +adopt pip's [VCS support](https://pip.pypa.io/en/stable/topics/vcs-support/) |
| 173 | +syntax. The URL `git+https://git.example/viking/viking.git@v1.1.0` clones the |
| 174 | +git repository over HTTPS and checks out the tag `v1.1.0`. |
| 175 | + |
| 176 | +### Matcher factory |
| 177 | + |
| 178 | +The matcher factory argument is an import string. The string must resolve to |
| 179 | +a callable that accepts a `ctx` argument and returns a `re.Pattern` |
| 180 | +(recommended) or `MatchFunction`. If the return value is a pattern object, |
| 181 | +then it must have exactly one match group. The pattern is matched with |
| 182 | +`re.match`. |
| 183 | + |
| 184 | +The default matcher factory parsed the tag with `packaging.version.Version` |
| 185 | +and ignores any error. Fromager will provide additional matcher factories for |
| 186 | +common tag patterns like `v1.2`, `1.2`, and `v1.2-stable`. |
| 187 | + |
| 188 | +```python |
| 189 | +import re |
| 190 | +
|
| 191 | +from fromager import context, resolver |
| 192 | +from packaging.version import Version |
| 193 | +
|
| 194 | +
|
| 195 | +def matcher_factory_pat(ctx: context.WorkContext) -> re.Pattern | resolver.MatchFunction: |
| 196 | + # tag must match 'v1.2+midstream.1.cpu' and results in Version("1.2+midstream.1") |
| 197 | + variant = re.escape(ctx.variant) |
| 198 | + pat = rf"^v(.*\+midstream\.\d+)\.{variant}$" |
| 199 | + return re.compile(pat) |
| 200 | +
|
| 201 | +
|
| 202 | +def matcher_factory_func(ctx: context.WorkContext) -> re.Pattern | resolver.MatchFunction: |
| 203 | + def pep440_matcher(identifier: str, item: str) -> Version | None: |
| 204 | + try: |
| 205 | + return Version(item) |
| 206 | + except ValueError: |
| 207 | + return None |
| 208 | + return pep440_matcher |
| 209 | +``` |
| 210 | + |
| 211 | +### Deprecations |
| 212 | + |
| 213 | +- `download_source.url` is handled by `pypi-download` profile or |
| 214 | + `release_artifact` parameter of `github` or `gitlab` provider |
| 215 | +- `download_source.destination_filename` is not needed. All sdists use |
| 216 | + standard `{dist_name}-{version}.tar.gz` file name |
| 217 | +- `resolver_dist.sdist_server_url` is replaced by `index_url` parameter. |
| 218 | + All `pypi-*` profile support a custom index. |
| 219 | +- `git_options.submodules` is not needed. Like pip, Fromager will always |
| 220 | + clone all submodules. |
| 221 | +- variant settings `wheel_server_url` and `pre_build` are replaced by |
| 222 | + `pypi-prebuilt` profile |
| 223 | + |
| 224 | +### Migration |
| 225 | + |
| 226 | +Top-level and variant-specific `source` settings are mutually exclusive with |
| 227 | +`download_source`, `resolver_dist`, `wheel_server_url`, and `pre_build` |
| 228 | +settings. It is an error to combine the new `source` settings with any of the |
| 229 | +old settings. |
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?v=4&size=40){kind=avatar}
0 commit comments