fix(sources): prepare Git checkouts by download kind

docs/concepts/bootstrapper-architecture.rst

@@ -57,7 +57,8 @@ the contract every phase must satisfy:
    * - ``Start``
      - Add to dependency graph; deduplicate already-seen packages
    * - ``PrepareSource``
-     - Download source or prebuilt wheel; read build-system deps
+     - Download source or prebuilt wheel; prepare archives or checkouts;
+       read build-system deps
    * - ``PrepareBuild``
      - Install build-system deps; discover backend and sdist deps
    * - ``Build``
@@ -67,6 +68,10 @@ the contract every phase must satisfy:
    * - ``Complete``
      - Clean up build directories (terminal phase)
 
+.. versionchanged:: 0.102.0
+   ``PrepareSource`` passes the downloaded artifact kind through preparation,
+   so Git checkouts can be prepared without unpacking them.
+
 Each phase wraps a ``WorkItem`` dataclass that accumulates per-package
 state (resolved version, build environment, build result, etc.) as it
 moves through the pipeline.

docs/concepts/hooks-and-overrides.rst

@@ -62,6 +62,12 @@ arguments.
    download results with explicit artifact kinds. See :ref:`download_source_hook`
    for the contract, legacy path defaults, and a migration example.
 
+.. versionchanged:: 0.102.0
+
+   ``prepare_source`` selects an archive or Git default using the downloaded
+   artifact kind. Package overrides remain active. See :ref:`prepare_source_hook`
+   for their arguments and the public Git checkout helper.
+
 Global Hooks
 ------------
 

docs/concepts/resolver-architecture.rst

@@ -113,15 +113,34 @@ settings are used.
 Hook downloads accept explicit artifact kinds and adapt legacy paths using
 profile-specific defaults. If no download hook exists, the profile selects the
 standard sdist or wheel downloader and its destination directory. Hook errors
-propagate with context instead of triggering a fallback. The typed result makes
-the artifact kind available for source preparation; consuming it in preparation
-and integrating prebuilt detection into the bootstrapper are separate steps.
+propagate with context instead of triggering a fallback. In-process callers
+pass the downloaded path and kind to preparation: ``sdist`` and ``tarball``
+are unpacked, while ``git_checkout`` uses the checkout directory. The checkout
+uses the same double-nested work directory pattern as an unpacked archive;
+the directory names can differ for normalized package names.
+The default Git preparer rejects a checkout with local changes before applying
+patches, project overrides, and Rust vendoring. Git status does not prove that the
+checkout is the original clone or preserve it after preparation. Build metadata
+is written after preparation but is not used to choose the preparation path.
+With ``--no-cleanup``, downloading to an existing checkout fails and preserves
+its files for inspection. Enable cleanup or use a new work directory to
+download and prepare a fresh checkout. Git status cannot detect changes to
+patch or settings files outside the checkout. If preparation left the checkout
+clean, a manual second ``prepare-source`` call runs the preparation steps again.
+The separate ``step prepare-source`` invocation accepts ``--source-kind`` and
+``--source-path`` because it cannot retrieve the earlier Python download result.
+``step download-source-archive --output-format json`` exposes that result's
+actual path and kind without changing its default path-only output.
+Without ``--source-kind``, it keeps the legacy archive behavior.
+Integrating prebuilt detection into the bootstrapper remains a separate step.
 See :ref:`download_source_hook` for the return contract and compatibility rules.
 
 .. versionchanged:: 0.102.0
-   Configured source resolvers also select the sdist build method. Git checkout
-   resolvers use PEP 517 by default; hook sdist resolvers require an explicit
-   ``build_sdist: pep517`` or ``build_sdist: tarball``. PyPI sdist resolvers
+   Explicit ``DownloadKind.git_checkout`` results can be prepared without
+   unpacking an archive. Configured source resolvers also select the sdist
+   build method. Git checkout resolvers use PEP 517 by default; hook sdist
+   resolvers require an explicit ``build_sdist: pep517`` or
+   ``build_sdist: tarball``. PyPI sdist resolvers
    repack a tarball. Both configured strategies take precedence over an installed
    ``build_sdist`` override and log a warning when it is ignored. Legacy
    settings continue to use the override.

docs/reference/hooks.rst

@@ -298,17 +298,32 @@ Source hooks
     the configured hook profiles; packages using legacy settings still require
     a bare ``Path``.
 
-.. autofromagerhook:: default_prepare_source
-
-    The ``prepare_source()`` function is responsible for setting up a tree
-    of source files in a format that is ready to be built. The default
-    implementation unpacks the source archive and applies patches.
+.. _prepare_source_hook:
 
-    The arguments are the ``WorkContext``, the ``Requirement`` being
-    evaluated, the ``Path`` to the source archive, and the version.
+.. autofromagerhook:: default_prepare_source
 
-    The return value should be the ``Path`` to the root of the source tree,
-    ideally inside the ``ctx.work_dir`` directory.
+    The ``prepare_source()`` function sets up a source tree for building. It
+    selects ``default_prepare_source()`` for archives or
+    ``default_prepare_git_checkout()`` for a Git checkout according to the
+    downloaded ``DownloadKind``. Both defaults apply patches and other project
+    preparation.
+
+    In-process callers pass the ``path`` and ``kind`` from ``DownloadedSource``
+    to ``prepare_source()``. The Python function accepts a source ``Path`` and
+    optional ``download_kind``; path-only callers default to ``sdist``.
+    A configured ``source:`` resolver does not disable the package
+    ``prepare_source`` override. The override receives the ``WorkContext``,
+    ``Requirement``, source ``Path``, and version, but not ``download_kind``.
+    An archive-specific override can call ``sources.default_prepare_source()``;
+    a Git-specific override can call
+    ``sources.default_prepare_git_checkout()`` before its own changes.
+
+    The override returns the source root ``Path`` or a ``(Path, bool)`` pair.
+    Git checkouts must use the double-nested package/version layout under
+    ``ctx.work_dir`` so the later bootstrap and step commands can find them.
+
+    .. versionchanged:: 0.102.0
+       Passing ``DownloadKind.git_checkout`` prepares a clean Git checkout.
 
 .. autofromagerhook:: default_build_sdist
 
@@ -328,6 +343,8 @@ Source helper functions
 The following helper functions are available in the ``fromager.sources`` module
 for use in custom source hooks:
 
+.. autofunction:: fromager.sources.default_prepare_git_checkout
+
 .. autofunction:: fromager.sources.ensure_pkg_info
 
 .. autofunction:: fromager.sources.pep517_build_sdist

docs/using.md

@@ -256,9 +256,19 @@ package index and downloads it. It will be common to run this step
 with `pypi.org`, but for truly isolated and reproducible builds a
 private index server is more robust.
 
-The `step prepare-source` command unpacks the source archive
-downloaded from the previous step and applies any patches (refer to
-[customization](customization.md) for details about patching).
+The `step prepare-source` command unpacks a source archive and applies any
+patches (refer to [customization](customization.md) for details about patching).
+It defaults to the archive kind for existing scripts. For a Git checkout, pass
+`--source-kind git_checkout`. Pass `--source-path` when the downloaded source
+cannot be found at its standard location. Git checkouts must use the standard
+double-nested work directory layout so later build steps can find them. The
+download command's optional
+`--output-format json` prints both `path` and `kind` when a hook can return
+different artifact kinds. Its default output remains the path alone. Git
+preparation applies patches to the selected checkout itself, so use a
+disposable clone for `--source-path`. Git downloads do not reuse an existing
+checkout with `--no-cleanup`; the files remain available for inspection. Enable
+cleanup or use a new work directory to download it again.
 
 The `step prepare-build` command creates a virtualenv with the build
 dependencies for building the wheel. It expects a `--wheel-server-url`

src/fromager/bootstrapper/_bootstrapper.py

@@ -482,28 +482,29 @@ def _download_source(
         req: Requirement,
         resolved_version: Version,
         source_url: str,
-    ) -> pathlib.Path:
+    ) -> packagesettings.DownloadedSource:
         """Download source for a package."""
         result: packagesettings.DownloadedSource = sources.download_source(
             ctx=self.ctx,
             req=req,
             version=resolved_version,
             download_url=source_url,
         )
-        return result.path
+        return result
 
     def _prepare_source(
         self,
         req: Requirement,
         resolved_version: Version,
-        source_filename: pathlib.Path,
+        downloaded_source: packagesettings.DownloadedSource,
     ) -> pathlib.Path:
         """Prepare (unpack/patch) source for building."""
         result: pathlib.Path = sources.prepare_source(
             ctx=self.ctx,
             req=req,
-            source_filename=source_filename,
+            source_filename=downloaded_source.path,
             version=resolved_version,
+            download_kind=downloaded_source.kind,
         )
         return result
 

src/fromager/bootstrapper/_prepare_source.py

@@ -29,7 +29,7 @@ def _bg_prepare_source(
     resolved_version: Version,
     source_url: str,
 ) -> PreparedSourceData:
-    """Background-safe source download+unpack: no Bootstrapper state accessed."""
+    """Background-safe source download and preparation without Bootstrapper state."""
     # Thread-safe: _seen_requirements in the main thread prevents the same
     # package from being submitted to the thread pool more than once.
     # Paths from download_source() and prepare_source() include {name}-{version},
@@ -43,11 +43,15 @@ def _bg_prepare_source(
             sdist_root_dir=unpacked / unpacked.stem,
             cached_wheel_filename=cached_wheel,
         )
-    source_filename = sources.download_source(
+    downloaded_source = sources.download_source(
         ctx=ctx, req=req, version=resolved_version, download_url=source_url
-    ).path
+    )
     sdist_root_dir = sources.prepare_source(
-        ctx=ctx, req=req, source_filename=source_filename, version=resolved_version
+        ctx=ctx,
+        req=req,
+        source_filename=downloaded_source.path,
+        version=resolved_version,
+        download_kind=downloaded_source.kind,
     )
     return PreparedSourceData(sdist_root_dir=sdist_root_dir)
 

src/fromager/commands/build.py

@@ -384,25 +384,26 @@ def _build(
     # If we get here and still don't have a wheel filename, then we need to
     # build the wheel.
     if not wheel_filename:
-        source_filename = sources.download_source(
+        downloaded_source = sources.download_source(
             ctx=wkctx,
             req=req,
             version=resolved_version,
             download_url=source_download_url,
-        ).path
+        )
         logger.debug(
-            "saved sdist of version %s from %s to %s",
+            "saved source of version %s from %s to %s",
             resolved_version,
             source_download_url,
-            source_filename,
+            downloaded_source.path,
         )
 
         # Prepare source
         source_root_dir = sources.prepare_source(
             ctx=wkctx,
             req=req,
-            source_filename=source_filename,
+            source_filename=downloaded_source.path,
             version=resolved_version,
+            download_kind=downloaded_source.kind,
         )
 
         # Build environment

src/fromager/commands/step.py

@@ -1,3 +1,4 @@
+import json
 import logging
 import pathlib
 
@@ -10,6 +11,7 @@
     clickext,
     context,
     finders,
+    packagesettings,
     server,
     sources,
     wheels,
@@ -26,17 +28,25 @@ def step() -> None:
 
 
 @step.command()
+@click.option(
+    "--output-format",
+    type=click.Choice(["path", "json"]),
+    default="path",
+    show_default=True,
+    help="Print the path or a JSON object with path and artifact kind.",
+)
 @click.argument("dist_name")
 @click.argument("dist_version", type=clickext.PackageVersion())
 @click.argument("sdist_server_url")
 @click.pass_obj
 def download_source_archive(
     wkctx: context.WorkContext,
+    output_format: str,
     dist_name: str,
     dist_version: Version,
     sdist_server_url: str,
 ) -> None:
-    """download the source code archive for one version of one package
+    """download source for one version of one package
 
     DIST_NAME is the name of a distribution
 
@@ -50,22 +60,52 @@ def download_source_archive(
     source_url, version = sources.resolve_source(
         ctx=wkctx, req=req, sdist_server_url=sdist_server_url
     )
-    filename = sources.download_source(
+    downloaded_source = sources.download_source(
         ctx=wkctx,
         req=req,
         version=version,
         download_url=source_url,
-    ).path
+    )
     requirement_ctxvar.reset(token)
-    print(filename)
+    if output_format == "json":
+        print(
+            json.dumps(
+                {
+                    "path": str(downloaded_source.path),
+                    "kind": downloaded_source.kind.value,
+                }
+            )
+        )
+    else:
+        print(downloaded_source.path)
 
 
 @step.command()
+@click.option(
+    "--source-kind",
+    type=click.Choice(
+        [
+            packagesettings.DownloadKind.sdist.value,
+            packagesettings.DownloadKind.tarball.value,
+            packagesettings.DownloadKind.git_checkout.value,
+        ]
+    ),
+    default=packagesettings.DownloadKind.sdist.value,
+    show_default=True,
+    help="Kind returned by the source download step.",
+)
+@click.option(
+    "--source-path",
+    type=click.Path(exists=True, path_type=pathlib.Path),
+    help="Downloaded source path; Git checkouts must use the standard work directory layout.",
+)
 @click.argument("dist_name")
 @click.argument("dist_version", type=clickext.PackageVersion())
 @click.pass_obj
 def prepare_source(
     wkctx: context.WorkContext,
+    source_kind: str,
+    source_path: pathlib.Path | None,
     dist_name: str,
     dist_version: Version,
 ) -> None:
@@ -78,22 +118,35 @@ def prepare_source(
     """
     req = Requirement(f"{dist_name}=={dist_version}")
     token = requirement_ctxvar.set(req)
+    download_kind = packagesettings.DownloadKind(source_kind)
     sdists_downloads = pathlib.Path(wkctx.sdists_repo) / "downloads"
-    source_filename = finders.find_sdist(
-        wkctx, wkctx.sdists_downloads, req, str(dist_version)
+    checkout = (
+        wkctx.work_dir / f"{req.name}-{dist_version}" / f"{req.name}-{dist_version}"
     )
+    source_filename: pathlib.Path | None
+    if source_path is not None:
+        source_filename = source_path
+    elif download_kind is packagesettings.DownloadKind.git_checkout:
+        source_filename = checkout if checkout.is_dir() else None
+    else:
+        source_filename = finders.find_sdist(
+            wkctx, wkctx.sdists_downloads, req, str(dist_version)
+        )
     if source_filename is None:
+        if download_kind is packagesettings.DownloadKind.git_checkout:
+            raise RuntimeError(f"Cannot find Git checkout for {req} at {checkout}")
         dir_contents: list[str] = []
         for ext in ["*.tar.gz", "*.zip"]:
             dir_contents.extend(str(e) for e in wkctx.sdists_downloads.glob(ext))
         raise RuntimeError(
-            f"Cannot find sdist for {req.name} version {dist_version} in {sdists_downloads} among {dir_contents}"
+            f"Cannot find source for {req.name} version {dist_version} in {sdists_downloads} among {dir_contents}"
         )
     source_root_dir = sources.prepare_source(
         ctx=wkctx,
         req=req,
         source_filename=source_filename,
         version=dist_version,
+        download_kind=download_kind,
     )
     requirement_ctxvar.reset(token)
     print(source_root_dir)