From a9ca682b55eff30bf70554b30f5a364ec24dccdb Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 21:48:03 +0000 Subject: [PATCH] fix: fetch only new Drupal core commits on top of the seed cache MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Short Summary (TL;DR) Drupal core workspaces downloaded all of core's history on every start, and a failed download left an empty checkout. This makes the seed cache's refs survive, so the fetch is a real delta and a failed fetch falls back to the cache. It also makes the hourly updater actually move the cache's branches. ## The Issue Drupal 11 and 12 integration jobs failed at "Verify workspace — web root exists", for example https://github.com/ddev/coder-ddev/actions/runs/44144337457. Each failure logged "Fetch from origin failed — continuing with cached refs" after 4 to 10 minutes. Drupal 12 then failed to install, and Drupal 11 failed to check out 11.x. Passing jobs also spent 80 to 514 seconds on that "delta" fetch. There were three causes: - The startup script fetched the cache into a `drupalcache` remote and then removed that remote. `git remote remove` also deletes the remote's refs, so the origin fetch had no refs to negotiate with and downloaded everything. - With no refs left, a failed origin fetch made `git checkout -b main --track origin/main` fail. Nothing checked its exit code, so setup reported "Drupal core cloned" with an empty working tree. - `git clone --bare` configures no fetch refspec. So `update-drupal-cache` ran `git fetch --all`, which only wrote `FETCH_HEAD` and left every branch where the first clone put it. ## How This PR Solves The Issue - `drupal-core/template.tf` fetches the cache's branches into `refs/remotes/origin/*` and its tags into `refs/tags/*`. The origin fetch now sends what the workspace already has. When the origin fetch fails, `main` and the version branches still check out from the cache. - A failed checkout of `main` now sets `SETUP_FAILED` with its own message. - `update-drupal-cache` fetches `+refs/heads/*:refs/heads/*` and `+refs/tags/*:refs/tags/*` into a bare cache. - `drupal-integration-test.yml` prints `SETUP_STATUS.txt` and the end of `/tmp/drupal-setup.log` when a job fails. Before this, these failures showed only "exit code 1". - The seed cache section of the server setup docs matches the new behavior. Its recovery steps now re-clone with `--bare`. ## Manual Testing Instructions On the Coder host, as the owner of the seed cache: ```bash git -C ~/cache/drupal-core-seed log -1 --format='%ci %h %s' main git ls-remote https://git.drupalcode.org/project/drupal.git refs/heads/main ``` Expect the cache to be behind upstream before the new updater runs. ```bash sudo install -m 755 drupal-core/scripts/update-drupal-cache /usr/local/bin/update-drupal-cache sudo systemctl start drupal-cache-updater.service git -C ~/cache/drupal-core-seed log -1 --format='%ci %h %s' main ``` Expect the cache's `main` to match upstream. Push this branch's drupal-core template, then create a Drupal 12 workspace. In `/tmp/drupal-setup.log`, expect "Fetched origin" to take seconds, not minutes, and `/home/coder/drupal-core/index.php` to exist. ## Automated Testing Overview Claude Code ran `terraform fmt -check -recursive`, `make validate`, and `make test-templates` on Linux, and all passed. Claude Code also checked the git commands against small local test repositories. The new fetch sends "have" lines, and with origin unreachable, `main` and `11.x` still check out from the cached refs. The real fetch time on staging needs CI to confirm. Nobody ran `make docs-check`. ## Release/Deployment Notes Reinstall `update-drupal-cache` to `/usr/local/bin` on every Coder host that has a seed cache, because the timer runs that installed copy. 🤖 Developed with assistance from [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01T3Cc2G6867sD8ZdDCVZ7Em --- .github/workflows/drupal-integration-test.yml | 13 ++++++ docs/content/self-hosting/server-setup.md | 13 +++--- drupal-core/scripts/update-drupal-cache | 13 +++++- drupal-core/template.tf | 41 +++++++++++++------ 4 files changed, 58 insertions(+), 22 deletions(-) diff --git a/.github/workflows/drupal-integration-test.yml b/.github/workflows/drupal-integration-test.yml index 6bb5ac4..9cbc7b5 100644 --- a/.github/workflows/drupal-integration-test.yml +++ b/.github/workflows/drupal-integration-test.yml @@ -235,6 +235,16 @@ jobs: fi echo "OK: anonymous request redirected to Coder login (site is private by default)" + - &dump-setup + # The checks above only report an exit code; the cause is in the + # workspace's setup log, which is gone once the workspace is deleted. + name: Dump setup status + if: failure() + run: | + coder ssh ${{ env.WORKSPACE_NAME }} -- cat /home/coder/SETUP_STATUS.txt || true + echo "--- last 60 lines of setup log ---" + coder ssh ${{ env.WORKSPACE_NAME }} -- tail -60 /tmp/drupal-setup.log || true + - &delete-workspace name: Delete workspace if: always() @@ -389,6 +399,7 @@ jobs: fi echo "OK: anonymous request reached the public Drupal site (drupal_site_public=true works)" + - *dump-setup - *delete-workspace - name: Release staging box lock @@ -576,6 +587,7 @@ jobs: fi echo "OK: anonymous request redirected to Coder login (site is private by default)" + - *dump-setup - name: Delete workspace if: always() run: coder delete ${{ env.WORKSPACE_NAME }} --yes || true @@ -778,6 +790,7 @@ jobs: fi echo "OK: anonymous request redirected to Coder login (site is private by default)" + - *dump-setup - name: Delete workspace if: always() run: coder delete ${{ env.WORKSPACE_NAME }} --yes || true diff --git a/docs/content/self-hosting/server-setup.md b/docs/content/self-hosting/server-setup.md index af1c8fa..22e064c 100644 --- a/docs/content/self-hosting/server-setup.md +++ b/docs/content/self-hosting/server-setup.md @@ -880,10 +880,9 @@ This deploys three templates: The `drupal-core` template can provision workspaces faster using a **seed cache** on the host. The cache is a **bare** git clone of drupal/drupal. When a workspace starts, the startup script: 1. Creates a fresh git repo in the workspace directory -2. Adds the bare cache as a local remote and fetches from it (fast — reuses local objects, no network) -3. Removes the cache remote -4. Fetches from `git.drupalcode.org` for any commits newer than the cache -5. Checks out `main` (or the issue branch) +2. Fetches the bare cache's branches and tags as `origin`'s (fast — reuses local objects, no network) +3. Fetches from `git.drupalcode.org` for any commits newer than the cache +4. Checks out `main` (or the issue branch), from the cache's refs if `git.drupalcode.org` can't be reached This avoids downloading hundreds of MB per workspace and eliminates working-tree issues that a non-bare cache causes. @@ -897,7 +896,7 @@ The seed directory IS the bare clone root — no subdirectory nesting. No workin ### Install the hourly update timer -The update script runs `git fetch --all --prune` to keep the bare cache current. Because there is no working tree, no `git merge` is needed. Install it as an hourly systemd timer: +The update script fetches every branch and tag into the bare cache to keep it current. Because there is no working tree, no `git merge` is needed. Install it as an hourly systemd timer: ```bash REPO=~/workspace/coder-ddev # adjust if your repo is elsewhere @@ -1023,10 +1022,10 @@ The `update-drupal-cache` script handles both bare and non-bare repos automatica **Update script fails:** ```bash -git -C ~/cache/drupal-core-seed fetch --all --prune +git -C ~/cache/drupal-core-seed fetch --prune origin '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*' # If this fails, check network connectivity or re-clone: rm -rf ~/cache/drupal-core-seed -git clone https://git.drupalcode.org/project/drupal.git ~/cache/drupal-core-seed +git clone --bare https://git.drupalcode.org/project/drupal.git ~/cache/drupal-core-seed ``` --- diff --git a/drupal-core/scripts/update-drupal-cache b/drupal-core/scripts/update-drupal-cache index 44b542f..243d317 100755 --- a/drupal-core/scripts/update-drupal-cache +++ b/drupal-core/scripts/update-drupal-cache @@ -9,7 +9,8 @@ # the real remote, avoiding several hundred MB of network transfer per workspace. # # Using a bare repo is intentional: there is no working tree to keep in sync and -# no "Your branch is behind" noise. git fetch --all --prune is all that is needed. +# no "Your branch is behind" noise. Branches and tags are fetched straight into +# refs/heads and refs/tags (a bare clone has no fetch refspec of its own). # # Initial setup (one-time, run as the seed-directory owner): # git clone --bare https://git.drupalcode.org/project/drupal.git ~/cache/drupal-core-seed @@ -70,7 +71,15 @@ echo "Started: $(date)" echo "" echo "Fetching Drupal core git objects..." -git -C "$SEED_DIR" fetch --all --prune +if [ "$IS_BARE" = "true" ]; then + # `git clone --bare` configures no fetch refspec, so a plain + # `git fetch --all` only writes FETCH_HEAD and leaves every branch where + # the initial clone put it. Fetch branches and tags into place explicitly. + git -C "$SEED_DIR" fetch --prune origin \ + '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*' +else + git -C "$SEED_DIR" fetch --all --prune +fi if [ "$IS_BARE" = "false" ]; then echo "Fast-forwarding local main branch..." diff --git a/drupal-core/template.tf b/drupal-core/template.tf index eed8102..5148ec2 100644 --- a/drupal-core/template.tf +++ b/drupal-core/template.tf @@ -552,13 +552,13 @@ STATUS_HEADER _t=$SECONDS DRUPAL_REMOTE="https://git.drupalcode.org/project/drupal.git" if git -C "$CACHE_SEED" rev-parse --is-bare-repository 2>/dev/null | grep -q true; then - # Bare cache: init repo, add all remotes (cache, origin, issue fork if - # applicable), fetch cache first (fast local copy of all objects), then - # fetch origin and issue fork for only the delta. + # Bare cache: init repo, add origin (and the issue fork if applicable), + # copy the cache's branches and tags in as origin's refs (fast local + # copy of all objects), then fetch origin and issue fork for only the + # delta. log_setup "Initialising Drupal core repo (bare cache + origin delta)..." mkdir -p "$DRUPAL_DIR" git -C "$DRUPAL_DIR" init -q >> "$SETUP_LOG" 2>&1 - git -C "$DRUPAL_DIR" remote add drupalcache "$CACHE_SEED" git -C "$DRUPAL_DIR" remote add origin "$DRUPAL_REMOTE" if [ "$USING_ISSUE_FORK" = "true" ] && [ -n "$ISSUE_FORK" ]; then # --no-tags: the fork's tags duplicate core's ~800 tags from origin. @@ -566,13 +566,22 @@ STATUS_HEADER fi # Fetches use -q so the log isn't flooded with one line per branch/tag # (core has ~800 refs); errors still go to stderr and land in the log. + # + # The cache's refs land directly under refs/remotes/origin/*, not under + # a separate cache remote. That matters twice: + # - The origin fetch below can only skip objects that local refs point + # to. Refs under a temporary remote vanish with `git remote remove`, + # which left nothing to negotiate with, so every workspace downloaded + # all of core's history again (3-10 minutes, and sometimes failed). + # - When the origin fetch fails, origin/main and the version branches + # still exist from the cache, so the checkout below works. _f=$SECONDS - git -C "$DRUPAL_DIR" fetch -q drupalcache >> "$SETUP_LOG" 2>&1 - log_setup " Fetched local cache ($((SECONDS - _f))s)" - # Remove the local cache remote now — objects are in the local repo and - # keeping it causes ambiguous-ref errors when checking out branches that - # exist in both drupalcache and origin. - git -C "$DRUPAL_DIR" remote remove drupalcache + if git -C "$DRUPAL_DIR" fetch -q --no-tags "$CACHE_SEED" \ + '+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' >> "$SETUP_LOG" 2>&1; then + log_setup " Fetched local cache ($((SECONDS - _f))s)" + else + log_setup " ⚠ Fetch from local cache failed — fetching everything from origin" + fi _f=$SECONDS if git -C "$DRUPAL_DIR" fetch -q origin >> "$SETUP_LOG" 2>&1; then log_setup " Fetched origin ($((SECONDS - _f))s)" @@ -587,7 +596,13 @@ STATUS_HEADER log_setup " ⚠ Fetch from issue fork failed" fi fi - git -C "$DRUPAL_DIR" checkout -b main --track origin/main >> "$SETUP_LOG" 2>&1 + if ! git -C "$DRUPAL_DIR" checkout -b main --track origin/main >> "$SETUP_LOG" 2>&1; then + # Without this, a failed checkout left an empty working tree that + # setup reported as cloned, and the failure only surfaced later as + # "Failed to install Drupal". + log_setup "✗ Failed to check out main (no origin/main from cache or origin)" + SETUP_FAILED=true + fi elif [ -d "$CACHE_SEED/.git" ]; then # Non-bare (legacy) cache: use as --reference hint to avoid re-downloading # objects that already exist locally. @@ -598,11 +613,11 @@ STATUS_HEADER log_setup "Cloning Drupal core..." git clone -q "$DRUPAL_REMOTE" "$DRUPAL_DIR" >> "$SETUP_LOG" 2>&1 fi - if [ -d "$DRUPAL_DIR/.git" ]; then + if [ "$SETUP_FAILED" != "true" ] && [ -d "$DRUPAL_DIR/.git" ]; then log_setup "✓ Drupal core cloned ($((SECONDS - _t))s)" update_status "✓ Git clone: Success" else - log_setup "✗ Failed to clone Drupal core" + [ -d "$DRUPAL_DIR/.git" ] || log_setup "✗ Failed to clone Drupal core" update_status "✗ Git clone: Failed" SETUP_FAILED=true fi