diff --git a/.automation/skills/upstream-rebase/SKILL.md b/.automation/skills/upstream-rebase/SKILL.md new file mode 100644 index 00000000000000..5f22a00da6d391 --- /dev/null +++ b/.automation/skills/upstream-rebase/SKILL.md @@ -0,0 +1,246 @@ +--- +name: upstream-rebase +description: Nightly rebase of edera/6.18-lts onto stable linux-6.18.y, run unattended in CI by .github/workflows/upstream-rebase.yml. Covers the replay, conflict resolution, build checks, the audit of upstream fixes that meet the series, and the report the workflow publishes. +--- + +# Nightly upstream rebase + +You are rebasing the Edera downstream kernel series (around 80 commits: +Hyper-V-nested-on-Xen, the Xen PV-IOMMU, NUMA-aware Xen backends, 2 MiB +ballooning, OpenPaX, and assorted fixes) on `edera/6.18-lts` onto the stable +`linux-6.18.y` branch. The goal is a replay that changes **nothing downstream**: every +difference in the final tree must come from the upstream delta alone. Where +that is impossible, because upstream and downstream touched the same code, you +resolve the conflict the way a careful kernel maintainer would and you say +exactly what you did. + +This runs unattended. Nobody is watching while you work, and nobody can answer +a question. Everything a reviewer needs has to end up in the report. + +## What happens to your result + +You cannot push, and nothing you write can cause the branch to be pushed. +Your report can only stop it. When you finish, the workflow independently: + +1. runs `verify-upstream-rebase.sh` against your branch, which checks from git + alone that the series sits on the upstream tip, is linear, lost or gained + no commit, and changed the tree by exactly the upstream delta; +2. builds x86_64 and arm64 `vmlinux` with `kernel-build.sh`, which turns on + every downstream feature as a built-in. + +If all of that passes **and your report's "Needs a decision" section is +`None.` with nothing marked UNSURE anywhere** (checked by +`report-needs-decision.sh`), the downstream branch is force-pushed to your +branch. Otherwise your branch goes up as a pull request with your report as +the body, and a maintainer reads it. + +So a question you raise is never lost to an automatic push. Raise one whenever +a maintainer should see something before the branch moves, even when the +rebase itself is clean, for example an upstream change that reaches a +downstream feature without touching the same lines. Do not use the word UNSURE +for anything else. + +So: a conflict you resolve **will** show up as drift and **will** go to a +human. That is the intended outcome, not a failure. Do not try to make a +resolution look clean, and never drop, squash, reorder or reword a downstream +commit to get past the checks. A rebase that reaches review with an honest +report is a success; one that hides a judgement call is not. + +## Inputs + +The workflow gives you, in the prompt: + +- `DOWNSTREAM`: the branch being rebased, `edera/6.18-lts`. +- `OLD_TIP`: its current commit (already checked out). +- `UPSTREAM`: the upstream commit to rebase onto, fetched as the local branch + `upstream-target`. +- `RESULT`: the local branch name your result must end up on. +- `REPORT`: the path your report must be written to. + +The automation's own scripts are in `.rebase-tools/scripts/`, copied from the +default branch. Use those, never a copy inside the tree you are rebasing: the +tree is what is under test. + +### How to run commands here + +Your shell commands are checked against an allowlist that matches the +command's first word. Shell variables, `$(...)`, `<(...)` and `VAR=x cmd` +prefixes will be refused, so in every snippet below `OLD_TIP`, `UPSTREAM`, +`RESULT`, `MB` and `<...>` are placeholders: substitute the literal SHA or +path, or a path you choose. Put scratch files, worktrees and build output +under `.rebase-scratch/` in the repository root (`mkdir -p .rebase-scratch`); +the kernel's `.gitignore` ignores it and nothing reads it after you. + +This is a blobless clone: file contents are fetched from GitHub the first time +a command needs them, so the first `git log -p` or checkout over a range is +slower than you expect. That is not a hang. + +## 1. Survey + +```sh +git merge-base OLD_TIP UPSTREAM # call the result MB +git rev-list --count --no-merges MB..OLD_TIP # downstream commits +git log --oneline --no-merges MB..UPSTREAM # what is new upstream +``` + +The upstream range is large (one or more stable releases, often hundreds of +commits each), so do not read it all. Find where it meets the +series: + +```sh +git diff --name-only --output=.rebase-scratch/down.files MB..OLD_TIP +git diff --name-only --output=.rebase-scratch/up.files MB..UPSTREAM +comm -12 .rebase-scratch/down.files .rebase-scratch/up.files # both sorted already +git log --oneline --no-merges MB..UPSTREAM -- +``` + +Read the messages and diffs of those upstream commits. They are where a +textually clean replay can still be semantically wrong: a helper downstream +calls that changed its locking or return convention, a struct field that +moved, a Kconfig symbol that was renamed. Note them; you come back to them in +step 4. + +The series may contain a merge commit from an old pull request. Rebasing +flattens it into its commits, which is expected and the checker accepts it; +do not use `--rebase-merges`. + +## 2. Rebase + +```sh +git switch -c RESULT OLD_TIP +git rebase --onto UPSTREAM MB RESULT +``` + +When a commit conflicts: + +- Read the upstream change that caused it **and** the downstream commit's + intent (its message, and the rest of its diff). Resolve so the downstream + commit does what it did before, on top of what upstream now does. +- Upstream wins on fixes. If a stable backport changed the code a downstream + commit edits, keep every check, lock, ordering constraint + and error path the fix introduced, and fit the downstream change around it. +- If upstream now contains the downstream change itself (it was upstreamed and + backported to stable), let the downstream commit go empty and let git drop + it. Record that, with the upstream commit. +- If you cannot tell what the right resolution is, do not guess silently. Make + the most conservative resolution you can defend, mark it **UNSURE** in the + report, and explain both readings. +- Never use `-X ours`, `-X theirs`, `--skip`, or `git checkout --ours/--theirs` + on a whole file to make a conflict go away. + +For each conflict, record: the downstream commit (subject), the files, the +upstream commit it collided with, and in a sentence or two what you kept and +why. + +## 3. Build + +The workflow re-runs these builds itself, but run them here so you can fix +what your resolutions broke. Each takes a while; run them once the rebase is +complete, not after every commit. + +```sh +bash .rebase-tools/scripts/kernel-build.sh x86_64 .rebase-scratch/obj-x86 4 +bash .rebase-tools/scripts/kernel-build.sh arm64 .rebase-scratch/obj-arm64 4 +``` + +Exit status 3 means a downstream option no longer survives `olddefconfig`: +upstream renamed or re-depended a Kconfig symbol the series relies on (or that +the build script names). Find out which, and say so in the report; if the +build script's list is what is stale, that is a **Needs a decision** item, +not something to work around. + +If a build fails because of the rebase (a resolution, or an upstream change a +downstream commit has not caught up with), fix it **in the downstream commit +that is wrong**. Interactive editors do not work here, so drive the todo list +with sed: `git -c sequence.editor="sed -i 's/^pick /edit /'" rebase +-i UPSTREAM`, amend, then `git rebase --continue`. Do not add a fix-up commit +on top unless there is no single commit it belongs to; if you must, its +subject starts with `rebase: ` and the report says why. + +If a build fails on the pristine upstream tree too, it is not yours: build +UPSTREAM the same way in a `.rebase-scratch/` worktree to confirm, and report +it as pre-existing. + +## 4. Audit what upstream brought in + +For every upstream commit flagged in step 1, check that its change **survived +the replay**. Ancestry is not enough: a downstream commit replayed on top can +edit the very lines a fix added. + +Every stable commit is a fix; those with a `CVE-` reference matter most. For +each one that touched a file the series also touches: + +```sh +git log --oneline UPSTREAM..RESULT -- +``` + +If that returns anything, read the **final tree**, not the commit graph: +confirm the fix's checks and ordering are intact and that downstream code +added after it is covered by them. Grep tree-wide for any function or flag +the fix deliberately removed, in case a downstream commit brought it back. + +Also look for semantic collisions that did not conflict: an upstream change +to a function's contract (locking, refcounting, return values, a new required +call) where a downstream commit calls that function. Each one you cannot rule +out is an UNSURE item. + +## 5. Check yourself + +Run the same checker the workflow will run, so the report can explain its +result rather than be surprised by it: + +```sh +bash .rebase-tools/scripts/verify-upstream-rebase.sh OLD_TIP RESULT UPSTREAM +``` + +Every drift line and every changed commit it lists must be accounted for in +your report by a conflict resolution, a dropped-empty commit or a build fix. +If one is not, you made a change you did not mean to: find it and undo it. + +## 6. Report + +Write the report to the `REPORT` path, in Markdown. It becomes a pull request +body when a human has to look, so lead with what they must decide. No +preamble, no sign-off. + +```markdown +## Summary +One paragraph: the tree, upstream range (N commits, short SHAs and the +`git describe` of each end), downstream commits replayed, conflicts resolved, +commits dropped, and whether you expect the checker to pass. + +## Needs a decision +One bullet per question, each starting **UNSURE:**, with both readings and +what you committed. "None." if there is nothing; anything else here stops the +automatic push. + +## Conflicts resolved +Per conflict: downstream commit, files, colliding upstream commit, what you +kept and why. "None." if none. + +## Dropped downstream commits +Subject and the upstream commit that made it empty. "None." if none. + +## Build fixes +What broke, which commit you fixed it in, why. "None." if none. + +## Upstream commits that meet the series +`short-sha subject` for each upstream commit that touched a file the series +touches, and for each fix among them your reading of the final tree. "None." +if none. + +## Builds +x86_64, arm64: pass, fail (with the first error), or pre-existing failure +(confirmed on pristine upstream). +``` + +Then stop. Leave RESULT checked out or not; the workflow reads the branch, +not the working tree. + +## If you cannot finish + +If the rebase cannot be completed (a conflict you cannot resolve at all, a +toolchain that will not run), run `git rebase --abort`, do not create the +RESULT branch, and write the report explaining where you stopped and why. The +workflow treats a missing RESULT branch as a failed night and opens an issue +with your report. diff --git a/.github/scripts/kernel-build.sh b/.github/scripts/kernel-build.sh new file mode 100755 index 00000000000000..2a4c38dd68cb80 --- /dev/null +++ b/.github/scripts/kernel-build.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# Builds the kernel the way the nightly automation judges a branch. +# +# defconfig alone compiles almost none of the Edera series: Hyper-V nesting, +# Xen dom0 and backends, the PV-IOMMU, OpenPaX and the NUMA-aware Xen paths +# are all off in it. This turns them on as built-ins and builds vmlinux, so a +# replay that breaks any of them, or links against a symbol upstream removed, +# fails here. Building only a subdirectory would miss those link errors. +# +# The rebase skill runs this same script before they finish, so +# what Claude checks and what the workflow checks cannot drift apart. +# +# Usage: +# kernel-build.sh [jobs] +# +# Run from the top of a kernel tree. is created if missing and may be +# reused between runs. For arm64 it uses CROSS_COMPILE (default +# aarch64-linux-gnu-) unless LLVM is set, in which case it builds with clang. +# +# Exit status: make's on a build failure; 3 if an option it asked for did not +# survive olddefconfig (a Kconfig dependency the series no longer satisfies). + +set -euo pipefail + +if [ $# -lt 2 ] || [ $# -gt 3 ]; then + echo "usage: $0 [jobs]" >&2 + exit 2 +fi +arch=$1 +objdir=$2 +jobs=${3:-$(nproc)} + +common=( + XEN XEN_BACKEND XEN_DOM0 XEN_BALLOON XEN_GNTDEV XEN_GRANT_DEV_ALLOC + NUMA ACPI_NUMA MEMORY_HOTPLUG MEMORY_HOTREMOVE ZONE_DEVICE + XEN_UNPOPULATED_ALLOC XEN_BACKEND_NUMA_AFFINITY + XEN_NETDEV_FRONTEND XEN_NETDEV_BACKEND XEN_BLKDEV_FRONTEND XEN_BLKDEV_BACKEND + NET_9P NET_9P_XEN 9P_FS + DRM DRM_VIRTIO_GPU FW_CFG_SYSFS + SECURITY OPENPAX OPENPAX_SOFTMODE OPENPAX_XATTR_PAX_FLAGS OPENPAX_MPROTECT + OPENPAX_EMUTRAMP +) +case $arch in +x86_64) + karch=x86 + # make reads LLVM from the environment; x86 is always the gcc build. + unset LLVM + cross=() + opts=( + "${common[@]}" + HYPERVISOR_GUEST PARAVIRT XEN_PV XEN_PVHVM_GUEST PCI_XEN + XEN_PCIDEV_FRONTEND XEN_PCIDEV_BACKEND XEN_IOMMU + HYPERV HYPERV_VMBUS HYPERV_NET PCI_HYPERV PCI_HYPERV_INTERFACE + ) + ;; +arm64) + karch=arm64 + if [ -n "${LLVM:-}" ]; then + cross=(LLVM=1) + else + cross=(CROSS_COMPILE="${CROSS_COMPILE:-aarch64-linux-gnu-}") + fi + opts=("${common[@]}") + ;; +*) + echo "unknown arch: $arch" >&2 + exit 2 + ;; +esac + +mkdir -p "$objdir" +mk() { make -s ARCH="$karch" "${cross[@]}" O="$objdir" "$@"; } + +mk defconfig +enable=() +for o in "${opts[@]}"; do enable+=(--enable "$o"); done +scripts/config --file "$objdir/.config" "${enable[@]}" +mk olddefconfig + +# scripts/config writes whatever it is told; olddefconfig then quietly drops +# anything whose dependencies are not met. Catch that, or a Kconfig change +# could switch a downstream feature off without failing the build. +lost=() +for o in "${opts[@]}"; do + grep -qx "CONFIG_$o=y" "$objdir/.config" || lost+=("$o") +done +if [ "${#lost[@]}" -ne 0 ]; then + echo "kernel-build: not enabled after olddefconfig ($arch): ${lost[*]}" >&2 + exit 3 +fi + +mk -j"$jobs" vmlinux +echo "kernel-build: $arch vmlinux built in $objdir" diff --git a/.github/scripts/report-needs-decision.sh b/.github/scripts/report-needs-decision.sh new file mode 100755 index 00000000000000..e15b51979fd3b7 --- /dev/null +++ b/.github/scripts/report-needs-decision.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Decides whether the nightly rebase report asks a human for a decision. +# +# The upstream-rebase skill tells Claude to list anything it was unsure about +# under a "## Needs a decision" section, each item marked UNSURE. A rebase can +# be clean by every mechanical measure (no drift, builds pass) and still raise +# such a question: an upstream change that interacts with a downstream feature +# without touching the same lines. The workflow does not push those +# automatically; it opens a pull request so a maintainer sees the question +# before the branch moves. +# +# The report counts as asking for a decision when either: +# - its "Needs a decision" section has any content other than "None.", or +# - any line marks something UNSURE. +# Both err towards a pull request: a stray "UNSURE" costs one review, a missed +# question costs an unreviewed kernel change. +# +# Usage: +# report-needs-decision.sh +# +# Prints the section's content when it asks for a decision. Exit status: +# 0 a decision is needed +# 1 no decision is needed +# 2 the report cannot be read + +set -euo pipefail + +if [ $# -ne 1 ] || [ ! -r "$1" ]; then + echo "usage: $0 " >&2 + exit 2 +fi + +awk ' + BEGIN { insec = 0; body = ""; unsure = 0 } + /UNSURE/ { unsure = 1 } + /^##?[ \t]/ { + if (tolower($0) ~ /^##?[ \t]+needs a decision[ \t]*$/) { insec = 1; next } + insec = 0 + } + insec { + line = $0 + gsub(/^[ \t]+|[ \t]+$/, "", line) + if (line == "") next + if (tolower(line) ~ /^(_?none\.?_?|n\/a\.?)$/) next + body = body $0 "\n" + } + END { + if (body != "" || unsure) { printf "%s", body; exit 0 } + exit 1 + } +' "$1" diff --git a/.github/scripts/tests/report-needs-decision.test.sh b/.github/scripts/tests/report-needs-decision.test.sh new file mode 100755 index 00000000000000..b2ab3547a2f1b6 --- /dev/null +++ b/.github/scripts/tests/report-needs-decision.test.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# Tests for .github/scripts/report-needs-decision.sh. +# +# A clean nightly rebase is force-pushed to its downstream branch unless this script says +# Claude's report asks for a decision, so a report that does ask must never +# read as one that does not. +# +# Run from anywhere: bash .github/scripts/tests/report-needs-decision.test.sh +set -uo pipefail + +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +SCRIPT="$HERE/../report-needs-decision.sh" + +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +failures=0 +expect() { + # expect (report on stdin) + local desc=$1 want=$2 got + cat >"$WORK/report.md" + bash "$SCRIPT" "$WORK/report.md" >/dev/null 2>&1 + got=$? + if [ "$got" -eq "$want" ]; then + echo "ok $desc" + else + echo "FAIL $desc: exit $got, want $want" >&2 + failures=$((failures + 1)) + fi +} + +expect "no section, nothing unsure" 1 <<'EOF' +## Summary +All 221 commits replayed cleanly. + +## Conflicts resolved +None. +EOF + +expect "section says None." 1 <<'EOF' +## Summary +Clean. + +## Needs a decision +None. + +## Conflicts resolved +None. +EOF + +expect "section empty before the next heading" 1 <<'EOF' +## Needs a decision + +## Conflicts resolved +None. +EOF + +expect "section with an item" 0 <<'EOF' +## Summary +Clean replay. + +## Needs a decision +- Upstream 0822a2e890 now reaches Arm dom0 vPCI; see below. + +## Conflicts resolved +None. +EOF + +expect "UNSURE outside the section" 0 <<'EOF' +## Summary +Clean replay. + +## Conflicts resolved +- foo.c: kept downstream's lock order. **UNSURE** whether upstream intended otherwise. +EOF + +expect "heading case and trailing space" 0 <<'EOF' +## NEEDS A DECISION +- something +EOF + +expect "a level-3 heading stays inside the section" 0 <<'EOF' +## Needs a decision +### Arm vPCI +- something +## Builds +pass +EOF + +expect "a report with one UNSURE item asks for a decision" 0 <<'EOF' +## Summary +Rebased cleanly. + +## Needs a decision +- **UNSURE: upstream `0123456789ab` ("xen/events: rework lateeoi handling") now reaches the downstream _on_node bind helpers.** + +## Conflicts resolved +None. +EOF + +bash "$SCRIPT" "$WORK/missing.md" >/dev/null 2>&1 +if [ $? -eq 2 ]; then echo "ok unreadable report is an error"; else + echo "FAIL unreadable report is an error" >&2 + failures=$((failures + 1)) +fi + +if [ "$failures" -ne 0 ]; then + echo "$failures failure(s)" >&2 + exit 1 +fi +echo "all passed" diff --git a/.github/scripts/tests/verify-upstream-rebase.test.sh b/.github/scripts/tests/verify-upstream-rebase.test.sh new file mode 100755 index 00000000000000..fdbdf7b5f86583 --- /dev/null +++ b/.github/scripts/tests/verify-upstream-rebase.test.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# Tests for .github/scripts/verify-upstream-rebase.sh. +# +# Builds a small throwaway repository with an upstream line and a downstream +# series on it, rebases the series onto a newer upstream, and then damages +# copies of that result in each way the checker exists to catch. The nightly +# rebase force-pushes edera/6.18-lts on this script's say-so, so every +# damaged copy must be refused, and every honest replay must pass. +# +# Run from anywhere: bash .github/scripts/tests/verify-upstream-rebase.test.sh +set -uo pipefail + +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +SCRIPT="$HERE/../verify-upstream-rebase.sh" + +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT +cd "$WORK" || exit 1 + +failures=0 +expect() { + # expect [] + local desc=$1 want=$2 got out + out=$(bash "$SCRIPT" "$3" "$4" "$5") + got=$? + if [ "$got" -ne "$want" ]; then + echo "FAIL $desc: exit $got, want $want" >&2 + printf ' %s\n' "${out//$'\n'/$'\n' }" >&2 + failures=$((failures + 1)) + elif [ $# -ge 6 ] && ! grep -qF -- "$6" <<<"$out"; then + echo "FAIL $desc: output lacks '$6'" >&2 + printf ' %s\n' "${out//$'\n'/$'\n' }" >&2 + failures=$((failures + 1)) + else + echo "ok $desc" + fi +} + +git init -q -b upstream . +git config user.name test +git config user.email test@example.invalid +commit() { git add -A && git commit -qm "$1"; } + +# Upstream: two files with room for both sides to edit. +seq 1 40 >a.c +seq 1 40 >b.c +commit "upstream: base" +base=$(git rev-parse HEAD) + +# Downstream series. Two commits share a subject, as real series do. +git switch -q -c downstream +sed -i 's/^10$/10 downstream/' a.c +commit "downstream: edit a" +sed -i 's/^30$/30 downstream/' b.c +commit "apply dep patch" +echo "new" >c.c +commit "downstream: add c" +sed -i 's/^35$/35 downstream/' b.c +commit "apply dep patch" +old=$(git rev-parse HEAD) + +# Upstream moves on in two steps. The first edits only lines far from +# anything downstream touches; the second edits line 12, which sits inside the +# hunk context of "downstream: edit a" (line 10) without conflicting with it. +git switch -q upstream +sed -i 's/^1$/1 upstream/' b.c +commit "upstream: edit b far away" +far=$(git rev-parse HEAD) +sed -i 's/^12$/12 upstream/' a.c +commit "upstream: edit a near downstream" +up=$(git rev-parse HEAD) + +replay() { # replay + git switch -q -c "$1" "$old" && git rebase -q --onto "$2" "$base" "$1" +} + +replay far "$far" +expect "replay with untouched context passes" 0 "$old" "$(git rev-parse HEAD)" "$far" "identical patch-ids" +expect "a no-op rebase passes" 0 "$old" "$old" "$base" "identical patch-ids" + +# Context moved but the commit's own lines did not: still clean. +replay rebased "$up" +good=$(git rev-parse HEAD) +expect "replay with moved context passes" 0 "$old" "$good" "$up" "1 changed only in hunk context" + +# Damage 1: one of the two same-subject commits edited. +git switch -q -c tamper "$good" +# shellcheck disable=SC2016 # expanded by the shell git rebase starts +git rebase -q --exec 'if [ "$(git log -1 --format=%s)" = "apply dep patch" ] && grep -q "35 downstream" b.c; then echo extra >>b.c; git commit -qa --amend --no-edit; fi' "$up" +expect "edited commit with a repeated subject fails" 1 "$old" "$(git rev-parse HEAD)" "$up" "1 commit(s) lost, added, or with different +/- lines" + +# Damage 2: a commit dropped. +git switch -q -c dropped "$good" +drop=$(git log --format='%H %s' "$up..HEAD" | awk '/downstream: add c/ { print $1 }') +git rebase -q --onto "$drop^" "$drop" dropped +expect "dropped commit fails" 1 "$old" "$(git rev-parse HEAD)" "$up" "downstream: add c" + +# Damage 3: a commit added. +git switch -q -c added "$good" +echo "sneaky" >d.c +commit "innocent looking" +expect "added commit fails" 1 "$old" "$(git rev-parse HEAD)" "$up" "innocent looking" + +# Damage 4: not on the upstream tip. +expect "stale base is refused" 2 "$old" "$old" "$up" "not based on the upstream tip" + +# Damage 5: a merge commit in the series. +git switch -q -c merged "$good" +git switch -q -c side "$up" +echo side >e.c +commit "side" +git switch -q merged +git merge -q --no-edit side +expect "merge in the series is refused" 2 "$old" "$(git rev-parse HEAD)" "$up" "merge commit" + +# A series that merged a feature branch, as both kernel trees once did: the +# first rebase flattens it, and that must still pass, since no commit is +# lost and the tree moves by exactly the upstream delta. +git switch -q -c withmerge "$old" +git switch -q -c feature "$old~1" +sed -i 's/^20$/20 feature/' a.c +commit "feature: edit a" +git switch -q withmerge +git merge -q --no-ff --no-edit feature >/dev/null +sed -i 's/^25$/25 after merge/' b.c +commit "downstream: after the merge" +mold=$(git rev-parse HEAD) +git switch -q -c flattened "$mold" +git rebase -q --onto "$up" "$base" flattened +expect "flattening a merged series passes" 0 "$mold" "$(git rev-parse HEAD)" "$up" "All 6 downstream commits replayed" + +if [ "$failures" -ne 0 ]; then + echo "$failures failure(s)" >&2 + exit 1 +fi +echo "all passed" diff --git a/.github/scripts/verify-upstream-rebase.sh b/.github/scripts/verify-upstream-rebase.sh new file mode 100755 index 00000000000000..7a493aad81c0c8 --- /dev/null +++ b/.github/scripts/verify-upstream-rebase.sh @@ -0,0 +1,207 @@ +#!/usr/bin/env bash +# Checks a rebased downstream branch against the branch it replaces, without +# trusting whoever did the rebase. +# +# The nightly upstream-rebase workflow lets a model drive the rebase, but +# nothing it says about its own result is taken on faith. This script is the +# gate. It proves, from git alone, that: +# +# 1. the new branch sits directly on the upstream tip it claims to; +# 2. the downstream series is linear (no merge commits slipped in); +# 3. no downstream commit was lost, added, or changed: patch-ids match, or, +# where they do not, the commit's own +/- lines are identical and only a +# hunk's context anchor moved; +# 4. the tree changed by exactly the upstream delta and nothing else (only +# the +/- lines are compared, so moved hunks and blob hashes do not raise +# false alarms). +# +# A rebase that needed conflict resolution will normally fail check 4: the +# resolution is, by definition, a change to downstream lines. That is the +# point. Such a result is not wrong, but it is not "clean", and a human has to +# read it before it replaces the branch. +# +# Usage: +# verify-upstream-rebase.sh +# +# Writes a Markdown report to stdout. Exit status: +# 0 clean: all four checks pass +# 1 drift: the branch is well-formed (1-2) but 3 or 4 found differences +# 2 broken: 1 or 2 failed, or the inputs are unusable + +set -euo pipefail + +if [ $# -ne 3 ]; then + echo "usage: $0 " >&2 + exit 2 +fi + +old=$(git rev-parse --verify "$1^{commit}") +new=$(git rev-parse --verify "$2^{commit}") +up=$(git rev-parse --verify "$3^{commit}") +old_base=$(git merge-base "$old" "$up") + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT + +status=0 +fail() { [ "$status" -ge "$1" ] || status=$1; } +short() { git rev-parse --short=12 "$1"; } +changed_lines() { grep -E '^[+-]' | grep -vE '^(\+\+\+|---)' || true; } +changed_files() { grep -E '^(\+\+\+|---) ' || true; } + +echo "| | |" +echo "| --- | --- |" +echo "| Old tip | \`$(short "$old")\` |" +echo "| New tip | \`$(short "$new")\` |" +echo "| Old upstream base | \`$(short "$old_base")\` |" +echo "| New upstream base | \`$(short "$up")\` |" +echo + +# 1. Base. +if [ "$(git merge-base "$new" "$up")" = "$up" ]; then + echo "- [x] New branch is based on the upstream tip." +else + echo "- [ ] **New branch is not based on the upstream tip** \`$(short "$up")\`." + fail 2 +fi + +# 2. Linearity. +merges=$(git rev-list --merges "$up..$new" | wc -l) +if [ "$merges" -eq 0 ]; then + echo "- [x] Downstream series is linear." +else + echo "- [ ] **Downstream series contains $merges merge commit(s).**" + fail 2 +fi + +if [ "$status" -ge 2 ]; then + echo + echo "Stopped: the remaining checks assume a linear series on the upstream tip." + exit "$status" +fi + +# 3. Commit set. +# +# Each commit is keyed by its subject and how many earlier commits in the +# series share that subject, so the k-th "apply dep patch" before pairs with +# the k-th after. A rebase keeps the series order, so this pairs every commit +# with its own replay even when subjects repeat. +# +# A changed patch-id is fine when the commit's own +/- lines are unchanged and +# only a hunk's context anchor moved, which is what happens wherever upstream +# edited the same region. Anything else is a real difference. +series() { + git log --no-merges --reverse --format='commit %H' --patch "$1" | + git patch-id --stable | awk '{ print $2 "\t" $1 }' >"$work/pid.map" + git log --no-merges --reverse --format='%H%x09%s' "$1" | + awk -F'\t' 'NR == FNR { pid[$1] = $2; next } + { k = $2 "#" ++seen[$2]; print k "\t" $1 "\t" ($1 in pid ? pid[$1] : "-") }' \ + "$work/pid.map" - +} +series "$old_base..$old" >"$work/old.series" +series "$up..$new" >"$work/new.series" +n_old=$(wc -l <"$work/old.series") +n_new=$(wc -l <"$work/new.series") + +# key -> "old-commit new-commit verdict"; verdict is same, check, lost, added. +awk -F'\t' 'NR == FNR { oc[$1] = $2; op[$1] = $3; order[++n] = $1; next } + { + if ($1 in oc) { print oc[$1], $2, (op[$1] == $3 ? "same" : "check"); delete oc[$1] } + else print "-", $2, "added" + } + END { for (i = 1; i <= n; i++) if (order[i] in oc) print oc[order[i]], "-", "lost" }' \ + "$work/old.series" "$work/new.series" >"$work/pairs" + +own_change() { + local patch + patch=$(git show --format= "$1") + changed_files <<<"$patch" + changed_lines <<<"$patch" +} +: >"$work/moved" +: >"$work/changed" +while read -r c d verdict; do + case $verdict in + same) ;; + check) + if [ "$(own_change "$c")" = "$(own_change "$d")" ]; then + echo "$c $d" >>"$work/moved" + else + echo "$c $d" >>"$work/changed" + fi ;; + *) echo "$c $d" >>"$work/changed" ;; + esac +done <"$work/pairs" + +if [ "$n_old" -eq "$n_new" ] && [ ! -s "$work/changed" ]; then + if [ -s "$work/moved" ]; then + echo "- [x] All $n_new downstream commits replayed;" \ + "$(wc -l <"$work/moved") changed only in hunk context (listed below)." + else + echo "- [x] All $n_new downstream commits replayed with identical patch-ids." + fi +else + echo "- [ ] **Downstream commit set changed:** $n_old before, $n_new after;" \ + "$(wc -l <"$work/changed") commit(s) lost, added, or with different +/- lines." + fail 1 +fi + +# 4. Drift. +git diff "$old..$new" >"$work/rebase.diff" +git diff "$old_base..$up" >"$work/upstream.diff" +changed_files <"$work/rebase.diff" >"$work/rebase.files" +changed_files <"$work/upstream.diff" >"$work/upstream.files" +changed_lines <"$work/rebase.diff" >"$work/rebase.lines" +changed_lines <"$work/upstream.diff" >"$work/upstream.lines" + +if cmp -s "$work/rebase.files" "$work/upstream.files" && + cmp -s "$work/rebase.lines" "$work/upstream.lines"; then + echo "- [x] Zero downstream drift: the tree changed by exactly the upstream delta." +else + echo "- [ ] **Downstream drift:** the tree changed by more than the upstream delta." + fail 1 +fi + +# Details for anything that did not match. +oneline() { if [ "$1" = - ]; then echo "(none)"; else git log -1 --format='%h %s' "$1"; fi; } +if [ -s "$work/moved" ]; then + echo + echo "
Commits whose hunk context moved (own +/- lines identical)" + echo + while read -r c d; do + echo "- \`$(oneline "$c")\` → \`$(git rev-parse --short "$d")\`" + done <"$work/moved" + echo + echo "
" +fi +if [ -s "$work/changed" ]; then + echo + echo "
Commits lost, added, or whose own +/- lines changed" + echo + echo "| Before | After |" + echo "| --- | --- |" + while read -r c d; do + echo "| \`$(oneline "$c")\` | \`$(oneline "$d")\` |" + done <"$work/changed" + echo + echo "
" +fi + +if ! cmp -s "$work/rebase.lines" "$work/upstream.lines" || + ! cmp -s "$work/rebase.files" "$work/upstream.files"; then + echo + echo "
Drift (changed lines: upstream delta vs. actual change)" + echo + echo '```diff' + { + diff -u --label upstream-files --label actual-files \ + "$work/upstream.files" "$work/rebase.files" || true + diff -u --label upstream-lines --label actual-lines \ + "$work/upstream.lines" "$work/rebase.lines" || true + } | head -n 400 + echo '```' + echo + echo "
" +fi + +exit "$status" diff --git a/.github/workflows/automation-selftest.yml b/.github/workflows/automation-selftest.yml new file mode 100644 index 00000000000000..b3cf554a636931 --- /dev/null +++ b/.github/workflows/automation-selftest.yml @@ -0,0 +1,60 @@ +name: Automation self-test + +# Tests the scripts the nightly rebase relies on to decide whether to +# force-push edera/6.18-lts, so a change that weakens either must not land +# quietly: +# +# - verify-upstream-rebase.sh: honest replays pass, including one that +# flattens a merged series; edited, dropped, added and merged commits and +# a stale base are all refused; +# - report-needs-decision.sh: any question in Claude's report, or anything +# marked UNSURE, stops the automatic push. +# +# It runs only when the automation itself changes. + +on: + pull_request: + types: [opened, synchronize, ready_for_review] + branches: ['edera/**'] + paths: + - '.github/scripts/**' + - '.github/workflows/upstream-rebase.yml' + - '.github/workflows/kernel-nightly.yml' + - '.github/workflows/rebase-land.yml' + - '.github/workflows/automation-selftest.yml' + - '.automation/**' + +permissions: + contents: read + +concurrency: + group: automation-selftest-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + selftest: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + # Only the automation is needed, not the kernel. + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + persist-credentials: false + sparse-checkout: | + .github + .automation + + - name: Rebase checker tests + run: bash .github/scripts/tests/verify-upstream-rebase.test.sh + + - name: Decision detector tests + run: bash .github/scripts/tests/report-needs-decision.test.sh + + - name: Shellcheck + run: shellcheck .github/scripts/*.sh .github/scripts/tests/*.sh diff --git a/.github/workflows/kernel-nightly.yml b/.github/workflows/kernel-nightly.yml new file mode 100644 index 00000000000000..f06f0c17c61c17 --- /dev/null +++ b/.github/workflows/kernel-nightly.yml @@ -0,0 +1,60 @@ +name: Nightly kernel maintenance + +# Rebases edera/6.18-lts onto stable linux-6.18.y every night +# (upstream-rebase.yml). +# +# Claude does the rebase. Independent checks decide whether the result is +# pushed or proposed as a pull request; see upstream-rebase.yml. +# +# GitHub runs scheduled workflows only from the default branch, edera/mainline, +# so this copy has no schedule of its own. lts-rebase-trigger.yml on +# edera/mainline dispatches it nightly; this file and the one on +# edera/mainline share a path, which is how that dispatch finds this copy. +# +# Repository configuration this needs: +# vars.REBASE_APP_CLIENT_ID, secrets.REBASE_APP_PRIVATE_KEY +# A GitHub App installed on this repository with contents, pull requests +# and workflows write, allowed to bypass the protection rules on +# edera/6.18-lts. GITHUB_TOKEN cannot do this job: it cannot push +# commits that touch .github/workflows/ (every rebase rewrites the ones +# that add them), and pull requests it opens do not trigger other +# workflows. +# vars.REBASE_FEDERATION_RULE_ID +# The workload identity federation rule for the rebase. It must accept +# this workflow's OIDC subject (a dispatched run on edera/6.18-lts). +# vars.ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID, +# ANTHROPIC_WORKSPACE_ID +# The rest of the Claude API federation identity. + +on: + workflow_dispatch: + inputs: + force: + description: Rebase even if an open rebase pull request already covers this upstream tip + type: boolean + default: false + +permissions: + contents: read + +concurrency: + # Per tree: groups are repository-wide, and edera/mainline's copy of this + # workflow uses plain kernel-nightly. + group: kernel-nightly-edera/6.18-lts + cancel-in-progress: false + +jobs: + lts: + uses: ./.github/workflows/upstream-rebase.yml + permissions: + contents: read + pull-requests: read + issues: write + id-token: write + with: + downstream: edera/6.18-lts + key: 6.18-lts + upstream_url: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git + upstream_branch: linux-6.18.y + force: ${{ inputs.force == true }} + secrets: inherit diff --git a/.github/workflows/rebase-land.yml b/.github/workflows/rebase-land.yml new file mode 100644 index 00000000000000..87eeda680ae715 --- /dev/null +++ b/.github/workflows/rebase-land.yml @@ -0,0 +1,130 @@ +name: Land upstream rebase + +# Lands a nightly rebase that needed review (see upstream-rebase.yml) once a +# maintainer approves its pull request. +# +# A rebase replaces the tree rather than merging into it, so the merge button +# is the wrong tool: it would graft a second copy of the whole series onto the +# old one. Approving instead force-pushes the tree to the exact commit that +# was approved, and GitHub then marks the pull request merged. +# +# A review runs this workflow from the pull request's merge ref, which is why +# it lives on edera/6.18-lts. edera/mainline carries its own copy for its +# own rebases. +# +# What has to hold before anything is pushed: +# - the pull request is a rebase/edera-6.18-lts/... branch from this +# repository, targeting edera/6.18-lts; +# - the review approved the current head, not an earlier one; +# - the approver has write access; +# - the tree is still the tip the branch was made from. The branch name +# records that tip, and branch names cannot be edited on an open pull +# request, so this is not something the pull request body can change. If +# the tree moved, nothing is pushed; the next nightly run starts from the +# new tip. + +on: + pull_request_review: + types: [submitted] + +permissions: + contents: read + +concurrency: + # Shared with the nightly's publish jobs, so nothing else moves the tree + # while this runs. + group: push-${{ github.event.pull_request.base.ref }} + cancel-in-progress: false + +jobs: + land: + if: >- + github.event.review.state == 'approved' + && github.event.pull_request.head.repo.full_name == github.repository + && github.event.pull_request.base.ref == 'edera/6.18-lts' + && startsWith(github.event.pull_request.head.ref, 'rebase/edera-6.18-lts/') + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: read + pull-requests: write + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: App token + id: app + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + with: + client-id: ${{ vars.REBASE_APP_CLIENT_ID }} + private-key: ${{ secrets.REBASE_APP_PRIVATE_KEY }} + permission-contents: write + permission-pull-requests: write + permission-workflows: write + + - name: Land + env: + APP_TOKEN: ${{ steps.app.outputs.token }} + JOB_TOKEN: ${{ github.token }} + PR: ${{ github.event.pull_request.number }} + BASE: ${{ github.event.pull_request.base.ref }} + HEAD_REF: ${{ github.event.pull_request.head.ref }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + REVIEWED_SHA: ${{ github.event.review.commit_id }} + APPROVER: ${{ github.event.review.user.login }} + run: | + set -euo pipefail + say() { + GH_TOKEN=$JOB_TOKEN gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --body "$1" + } + + if [ "$REVIEWED_SHA" != "$HEAD_SHA" ]; then + say "Not landed: the approval was for \`${REVIEWED_SHA:0:12}\`, but the head is now \`${HEAD_SHA:0:12}\`. Approve the current head to land it." + exit 1 + fi + + perm=$(GH_TOKEN=$JOB_TOKEN gh api "repos/$GITHUB_REPOSITORY/collaborators/$APPROVER/permission" -q .permission) + case $perm in + admin|maintain|write) ;; + *) + say "Not landed: @$APPROVER does not have write access, so their approval cannot land a rebase." + exit 1 ;; + esac + + # Name: rebase//--. + rest=${HEAD_REF#"rebase/${BASE//\//-}/"} + IFS=- read -r _date old12 up12 extra <<<"$rest" + if [ -n "${extra:-}" ] || ! [[ $old12 =~ ^[0-9a-f]{12}$ && $up12 =~ ^[0-9a-f]{12}$ ]]; then + say "Not landed: \`$HEAD_REF\` is not a branch the nightly automation made." + exit 1 + fi + + origin="https://x-access-token:${APP_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" + cur=$(git ls-remote "$origin" "refs/heads/$BASE" | cut -f1) + if [ "${cur:0:12}" != "$old12" ]; then + say "Not landed: \`$BASE\` has moved to \`${cur:0:12}\` since this branch was made from \`$old12\`. Tonight's run starts from the new tip; this pull request will be closed as superseded." + exit 1 + fi + + # Both commits are already on GitHub, so this needs neither the + # kernel's history nor its files, only the two commit objects to + # name in the push. + git init -q work && cd work + git fetch -q --no-tags --depth=1 --filter=tree:0 "$origin" "$HEAD_SHA" "$cur" + + # Created through the API, and before the tree moves: a push from + # this shallow clone is refused once the old tip is unreferenced. + backup="backup/${BASE//\//-}-pre-rebase-$(date -u +%Y%m%d)-$old12" + GH_TOKEN=$APP_TOKEN gh api -X POST "repos/$GITHUB_REPOSITORY/git/refs" \ + -f ref="refs/heads/$backup" -f sha="$cur" >/dev/null + + git push --force-with-lease="refs/heads/$BASE:$cur" "$origin" "$HEAD_SHA:refs/heads/$BASE" + now=$(git ls-remote "$origin" "refs/heads/$BASE" | cut -f1) + if [ "$now" != "$HEAD_SHA" ]; then + say "Push reported success but \`$BASE\` is \`${now:0:12}\`, not \`${HEAD_SHA:0:12}\`. Check the run log." + exit 1 + fi + say "Landed by @$APPROVER's approval: \`$BASE\` is now \`${HEAD_SHA:0:12}\` (was \`${cur:0:12}\`, kept as \`$backup\`)." + git push "$origin" --delete "$HEAD_REF" || true diff --git a/.github/workflows/upstream-rebase.yml b/.github/workflows/upstream-rebase.yml new file mode 100644 index 00000000000000..7b8620fb30878c --- /dev/null +++ b/.github/workflows/upstream-rebase.yml @@ -0,0 +1,665 @@ +name: Upstream rebase + +# Rebases an Edera kernel integration tree onto its upstream. Called nightly +# by kernel-nightly.yml for edera/6.18-lts, onto stable linux-6.18.y. +# +# Claude does the rebase, following .automation/skills/upstream-rebase/ +# SKILL.md: it replays the series, resolves conflicts, fixes what the replay +# broke, builds, audits upstream fixes that meet the series, and writes a +# report. +# +# Nothing Claude says about its own result decides what happens to it. The +# model runs in a job that holds a read-only token and cannot push. Its branch +# leaves that job as a git bundle, and two things then judge it +# independently, with scripts taken from the commit this run started from +# (the default branch), never from the tree under test: +# +# - verify-upstream-rebase.sh proves the series sits on the upstream tip, is +# linear, lost or gained no commit, and changed the tree by exactly the +# upstream delta; +# - kernel-build.sh builds x86_64 and arm64 vmlinux with every downstream +# feature built in. +# +# If both pass and Claude's report asks for no decision +# (report-needs-decision.sh), the tree is force-pushed to the result, pinned +# by lease to the tip the rebase started from, with the old tip kept as a +# backup branch. Otherwise, or if that push is refused, the result goes up as +# a rebase//... branch with a pull request carrying Claude's report; +# approving that pull request lands it (rebase-land.yml). A night where +# Claude could not produce a branch at all opens an issue instead. +# +# Nights with nothing new upstream stop in the first job and never start the +# model. +# +# Every job clones with --filter=blob:none: the kernel's full history with +# blobs is several gigabytes, and the jobs need the commit graph but only the +# contents of the files the series and the upstream delta touch. + +on: + workflow_call: + inputs: + downstream: + description: The tree to rebase, e.g. edera/6.18-lts + type: string + required: true + key: + description: A short name for the tree, unique per run, for artifact names + type: string + required: true + upstream_url: + type: string + required: true + upstream_branch: + type: string + required: true + force: + description: Rebase even if an open rebase pull request already covers this upstream tip + type: boolean + default: false + +permissions: + contents: read + +env: + DOWNSTREAM: ${{ inputs.downstream }} + UPSTREAM_URL: ${{ inputs.upstream_url }} + UPSTREAM_BRANCH: ${{ inputs.upstream_branch }} + +jobs: + prepare: + runs-on: ubuntu-latest + timeout-minutes: 30 + permissions: + contents: read + pull-requests: read + outputs: + run: ${{ steps.plan.outputs.run }} + old: ${{ steps.plan.outputs.old }} + up: ${{ steps.plan.outputs.up }} + prefix: ${{ steps.plan.outputs.prefix }} + candidate: ${{ steps.plan.outputs.candidate }} + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ env.DOWNSTREAM }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + - name: Decide whether tonight has work + id: plan + env: + GH_TOKEN: ${{ github.token }} + FORCE: ${{ inputs.force }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + old=$(git rev-parse HEAD) + up=$(git rev-parse FETCH_HEAD) + prefix="rebase/${DOWNSTREAM//\//-}/" + candidate="${prefix}$(date -u +%Y%m%d)-${old:0:12}-${up:0:12}" + { + echo "old=$old" + echo "up=$up" + echo "prefix=$prefix" + echo "candidate=$candidate" + } >> "$GITHUB_OUTPUT" + + skip() { + echo "run=false" >> "$GITHUB_OUTPUT" + echo "### Upstream rebase of \`$DOWNSTREAM\`" >> "$GITHUB_STEP_SUMMARY" + echo "$1" >> "$GITHUB_STEP_SUMMARY" + } + + if git merge-base --is-ancestor "$up" "$old"; then + skip "Nothing to do: \`$DOWNSTREAM\` (\`${old:0:12}\`) already contains \`$UPSTREAM_BRANCH\` (\`${up:0:12}\`)." + exit 0 + fi + + # The candidate name records both tips, so an open pull request for + # the same pair means last night's result is still awaiting review. + if [ "$FORCE" != "true" ]; then + pending=$(gh pr list --repo "$GITHUB_REPOSITORY" --state open --base "$DOWNSTREAM" \ + --json headRefName,url \ + -q ".[] | select(.headRefName | startswith(\"$prefix\") and endswith(\"-${old:0:12}-${up:0:12}\")) | .url") + if [ -n "$pending" ]; then + skip "A rebase of these exact tips is already awaiting review: $pending" + exit 0 + fi + fi + echo "run=true" >> "$GITHUB_OUTPUT" + + rebase: + needs: prepare + if: needs.prepare.outputs.run == 'true' + runs-on: ubuntu-latest + timeout-minutes: 330 + # Read-only on purpose: the model runs here. It reaches the Claude API + # through OIDC federation, and that is the only thing id-token is for. + permissions: + contents: read + id-token: write + outputs: + produced: ${{ steps.collect.outputs.produced }} + result: ${{ steps.collect.outputs.result }} + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + # The skill and scripts come from the commit this run started from, + # never from the tree under test. The kernel's .gitignore ignores + # dot-directories, so .rebase-tools/ cannot end up in a commit. + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir .rebase-tools + git archive "$GITHUB_SHA" .github/scripts .automation | tar -x -C .rebase-tools --strip-components=1 + + - name: Fetch upstream + env: + UP: ${{ needs.prepare.outputs.up }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git branch upstream-target "$UP" + # The rebase commits; this identity is what the committer field shows. + git config --global user.name "edera-upstream-rebase" + git config --global user.email "noreply@edera.dev" + + - name: Install toolchains + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends \ + build-essential bc bison flex libelf-dev libssl-dev python3 \ + gcc-aarch64-linux-gnu + + - name: Normalize identifiers + id: ids + env: + RULE: ${{ vars.REBASE_FEDERATION_RULE_ID }} + ORG: ${{ vars.ANTHROPIC_ORGANIZATION_ID }} + SVC: ${{ vars.ANTHROPIC_SERVICE_ACCOUNT_ID }} + WS: ${{ vars.ANTHROPIC_WORKSPACE_ID }} + run: | + set -euo pipefail + strip() { printf '%s' "$1" | tr -d '[:space:]'; } + { + echo "rule=$(strip "$RULE")" + echo "org=$(strip "$ORG")" + echo "svc=$(strip "$SVC")" + echo "ws=$(strip "$WS")" + } >> "$GITHUB_OUTPUT" + # An unset variable otherwise surfaces as an opaque API failure a + # second into the model step, with nothing in the log to say why. + missing=0 + for n in REBASE_FEDERATION_RULE_ID:RULE ANTHROPIC_ORGANIZATION_ID:ORG ANTHROPIC_SERVICE_ACCOUNT_ID:SVC ANTHROPIC_WORKSPACE_ID:WS; do + eval "v=\$${n#*:}" + if [ -z "$(strip "$v")" ]; then + echo "::error::vars.${n%%:*} is empty or not visible to this repository." + missing=1 + fi + done + exit "$missing" + + - name: Rebase + id: claude + continue-on-error: true + uses: anthropics/claude-code-action@6fed3ca145920b639991cb756090506e1bcaf515 # v1.0.245 + with: + # The job token, which is read-only here. Without it the action + # would exchange OIDC for its own app token, which can write. + github_token: ${{ github.token }} + anthropic_federation_rule_id: '${{ steps.ids.outputs.rule }}' + anthropic_organization_id: '${{ steps.ids.outputs.org }}' + anthropic_service_account_id: '${{ steps.ids.outputs.svc }}' + anthropic_workspace_id: '${{ steps.ids.outputs.ws }}' + prompt: | + Rebase the downstream kernel series onto upstream by following + .rebase-tools/skills/upstream-rebase/SKILL.md exactly. Read it in + full before you start. + + DOWNSTREAM=${{ env.DOWNSTREAM }} + OLD_TIP=${{ needs.prepare.outputs.old }} (${{ env.DOWNSTREAM }}, checked out) + UPSTREAM=${{ needs.prepare.outputs.up }} (${{ env.UPSTREAM_BRANCH }}, local branch upstream-target) + RESULT=rebase-result + REPORT=${{ runner.temp }}/rebase-report.md + + You cannot push and must not try. The workflow verifies and + builds your branch independently afterwards and decides what + happens to it. + claude_args: | + --model claude-opus-5-5 + --max-turns 400 + --allowedTools "Read,Grep,Glob,Edit,Write,Bash(git:*),Bash(make:*),Bash(bash .rebase-tools/scripts/verify-upstream-rebase.sh:*),Bash(bash .rebase-tools/scripts/kernel-build.sh:*),Bash(scripts/config:*),Bash(mkdir -p .rebase-scratch:*),Bash(diff:*),Bash(comm:*),Bash(sort:*),Bash(grep:*),Bash(sed:*),Bash(awk:*),Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(wc:*),Bash(ls:*),Bash(cut:*),Bash(tee:*)" + + # Take only what Claude left in git and the report file. A branch that + # does not exist or does not sit on the upstream tip is a failed night. + - name: Collect the result + id: collect + if: always() + env: + REPORT: ${{ runner.temp }}/rebase-report.md + UP: ${{ needs.prepare.outputs.up }} + OUTCOME: ${{ steps.claude.outcome }} + EXECUTION_FILE: ${{ steps.claude.outputs.execution_file }} + run: | + set -euo pipefail + out=$RUNNER_TEMP/out + mkdir -p "$out" + if [ -s "$REPORT" ]; then + cp "$REPORT" "$out/report.md" + else + # The action hides the session to keep tool output out of a + # public log, which also hides why a run ended early. Take only + # the API error messages and the final result text from it: + # neither carries tool output. + why= + f=${EXECUTION_FILE:-$RUNNER_TEMP/claude-execution-output.json} + if [ -s "$f" ]; then + why=$(jq -r ' + (if type == "array" then .[] else . end) + | if .type == "assistant" and (.error != null) then + "API error (\(.error)): " + ([.message.content[]? | .text? // empty] | join(" ")) + elif .type == "result" then + ([.result // empty] + (.errors // [] | map(tostring))) | join("\n") + else empty end' "$f" 2>/dev/null | head -c 2000 || true) + fi + { + printf 'Claude left no report (rebase step: %s).\n' "$OUTCOME" + if [ -n "$why" ]; then + # shellcheck disable=SC2016 # a literal Markdown fence + printf '\n```\n%s\n```\n' "$why" + else + printf 'The run log has the cause.\n' + fi + } >"$out/report.md" + [ -z "$why" ] || echo "::error::Claude ended without a result: ${why//$'\n'/ }" + fi + if git rev-parse --verify -q refs/heads/rebase-result >/dev/null \ + && git merge-base --is-ancestor "$UP" rebase-result \ + && [ "$(git rev-parse rebase-result)" != "$UP" ]; then + git bundle create "$out/rebase.bundle" refs/heads/rebase-result "^$UP" + echo "produced=true" >> "$GITHUB_OUTPUT" + echo "result=$(git rev-parse rebase-result)" >> "$GITHUB_OUTPUT" + else + echo "produced=false" >> "$GITHUB_OUTPUT" + fi + + - name: Upload + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/out/ + if-no-files-found: error + retention-days: 30 + + verify: + needs: [prepare, rebase] + if: needs.rebase.outputs.produced == 'true' + runs-on: ubuntu-latest + timeout-minutes: 45 + outputs: + status: ${{ steps.check.outputs.status }} + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + # The checker comes from the commit this run started from, never from + # the branch under test. + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir "$RUNNER_TEMP/tools" + git archive "$GITHUB_SHA" .github/scripts | tar -x -C "$RUNNER_TEMP/tools" --strip-components=1 + + - name: Download + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/in + + - name: Check + id: check + env: + OLD: ${{ needs.prepare.outputs.old }} + UP: ${{ needs.prepare.outputs.up }} + RESULT: ${{ needs.rebase.outputs.result }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git fetch "$RUNNER_TEMP/in/rebase.bundle" rebase-result:rebase-result + [ "$(git rev-parse rebase-result)" = "$RESULT" ] || { echo "bundle does not hold $RESULT"; exit 1; } + # Actions runs this step under bash -e, so a bare call would end the + # step on the checker's first non-zero exit, before its verdict is + # recorded: drift would then reach the pull request as "the checker + # did not run". Capture the status instead. + rc=0 + bash "$RUNNER_TEMP/tools/scripts/verify-upstream-rebase.sh" "$OLD" "$RESULT" "$UP" >"$RUNNER_TEMP/verify.md" || rc=$? + case $rc in + 0) status=clean ;; + 1) status=drift ;; + *) status=broken ;; + esac + echo "status=$status" >> "$GITHUB_OUTPUT" + cat "$RUNNER_TEMP/verify.md" >> "$GITHUB_STEP_SUMMARY" + + - name: Upload + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: verify-${{ inputs.key }} + path: ${{ runner.temp }}/verify.md + retention-days: 30 + + build: + needs: [prepare, rebase] + if: needs.rebase.outputs.produced == 'true' + runs-on: ubuntu-latest + timeout-minutes: 120 + strategy: + fail-fast: false + matrix: + arch: [x86_64, arm64] + name: build (${{ matrix.arch }}) + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir "$RUNNER_TEMP/tools" + git archive "$GITHUB_SHA" .github/scripts | tar -x -C "$RUNNER_TEMP/tools" --strip-components=1 + + - name: Download + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/in + + - name: Check out the result + env: + UP: ${{ needs.prepare.outputs.up }} + RESULT: ${{ needs.rebase.outputs.result }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git fetch "$RUNNER_TEMP/in/rebase.bundle" rebase-result:rebase-result + git checkout --detach "$RESULT" + + - name: Install toolchains + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends \ + build-essential bc bison flex libelf-dev libssl-dev python3 \ + gcc-aarch64-linux-gnu + + - name: Build + env: + ARCH: ${{ matrix.arch }} + run: bash "$RUNNER_TEMP/tools/scripts/kernel-build.sh" "$ARCH" "$RUNNER_TEMP/obj" + + publish: + needs: [prepare, rebase, verify, build] + if: always() && needs.prepare.outputs.run == 'true' && needs.rebase.result != 'cancelled' + runs-on: ubuntu-latest + timeout-minutes: 45 + concurrency: + group: push-${{ inputs.downstream }} + cancel-in-progress: false + permissions: + contents: read + issues: write + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir "$RUNNER_TEMP/tools" + git archive "$GITHUB_SHA" .github/scripts | tar -x -C "$RUNNER_TEMP/tools" --strip-components=1 + + - name: Download the rebase + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/in/rebase + + # Absent when Claude produced no branch and the checker never ran. + - name: Download the checker report + if: needs.verify.result == 'success' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: verify-${{ inputs.key }} + path: ${{ runner.temp }}/in/verify + + - name: App token + id: app + if: needs.rebase.outputs.produced == 'true' + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + with: + client-id: ${{ vars.REBASE_APP_CLIENT_ID }} + private-key: ${{ secrets.REBASE_APP_PRIVATE_KEY }} + permission-contents: write + permission-pull-requests: write + permission-workflows: write + + - name: Push or propose + env: + APP_TOKEN: ${{ steps.app.outputs.token }} + JOB_TOKEN: ${{ github.token }} + PRODUCED: ${{ needs.rebase.outputs.produced }} + VERIFY: ${{ needs.verify.outputs.status }} + BUILD: ${{ needs.build.result }} + OLD: ${{ needs.prepare.outputs.old }} + UP: ${{ needs.prepare.outputs.up }} + RESULT: ${{ needs.rebase.outputs.result }} + PREFIX: ${{ needs.prepare.outputs.prefix }} + CANDIDATE: ${{ needs.prepare.outputs.candidate }} + KEY: ${{ inputs.key }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + in=$RUNNER_TEMP/in + report=$in/rebase/report.md + verify=$in/verify/verify.md + summary() { cat >> "$GITHUB_STEP_SUMMARY"; } + title="Nightly upstream rebase of $DOWNSTREAM failed" + + # One issue per tree that stays open until someone deals with it, + # rather than one per night. + report_issue() { + existing=$(GH_TOKEN=$JOB_TOKEN gh issue list --repo "$GITHUB_REPOSITORY" --state open \ + --search "in:title \"$title\"" --json number -q '.[0].number // empty') + if [ -n "$existing" ]; then + GH_TOKEN=$JOB_TOKEN gh issue comment "$existing" --repo "$GITHUB_REPOSITORY" --body-file "$1" + else + GH_TOKEN=$JOB_TOKEN gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$1" + fi + } + + if [ "$PRODUCED" != "true" ]; then + body=$RUNNER_TEMP/issue.md + { + echo "Claude could not produce a rebase of \`$DOWNSTREAM\` (\`${OLD:0:12}\`) onto \`$UPSTREAM_BRANCH\` (\`${UP:0:12}\`). [Run log]($RUN_URL)." + echo + cat "$report" + } | head -c 60000 >"$body" + report_issue "$body" + echo "No branch produced; reported in an issue." | summary + exit 1 + fi + + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git fetch "$in/rebase/rebase.bundle" rebase-result:rebase-result + [ "$(git rev-parse rebase-result)" = "$RESULT" ] + + # Workflows triggered by a pull request, including the land workflow, + # run that branch's own copy of .github/ with this repository's + # secrets. So the automation paths must change by exactly what + # upstream changed there, or the branch never leaves this job: not + # as the tree and not as a pull request. + mb=$(git merge-base "$OLD" "$UP") + guarded=(.github .automation) + lines() { git diff "$@" -- "${guarded[@]}" | grep -E '^([+-]|diff )' || true; } + if [ "$(lines "$OLD" "$RESULT")" != "$(lines "$mb" "$UP")" ]; then + body=$RUNNER_TEMP/issue.md + { + echo "**Refused to publish** the rebase of \`$DOWNSTREAM\` (\`${OLD:0:12}\`) onto \`$UPSTREAM_BRANCH\` (\`${UP:0:12}\`): it changes \`${guarded[*]}\` beyond what upstream changed there. Those paths run with repository secrets, so the branch was not pushed anywhere. It is in the run's \`rebase-${KEY}\` artifact. [Run log]($RUN_URL)." + echo + echo '```diff' + git diff --stat "$OLD" "$RESULT" -- "${guarded[@]}" + echo '```' + } >"$body" + report_issue "$body" + echo "::error::Rebase changes automation paths beyond the upstream delta; not published." + exit 1 + fi + + origin="https://x-access-token:${APP_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" + export GH_TOKEN=$APP_TOKEN + + close_superseded() { + gh pr list --repo "$GITHUB_REPOSITORY" --state open --base "$DOWNSTREAM" \ + --json number,headRefName \ + -q ".[] | select(.headRefName | startswith(\"$PREFIX\")) | select(.headRefName != \"$CANDIDATE\") | .number" | + while read -r n; do + gh pr close "$n" --repo "$GITHUB_REPOSITORY" --delete-branch \ + --comment "Superseded by $1." + done + } + + # Reasons a human has to look before the tree moves. The checker + # and the builds judge the branch; the report can still raise a + # question they cannot see, such as an upstream change that meets a + # downstream feature without touching the same lines. + why=() + [ "$VERIFY" = "clean" ] || why+=("the checker found ${VERIFY:-no result (it did not run)}") + [ "$BUILD" = "success" ] || why+=("a build ${BUILD}") + drc=0 + bash "$RUNNER_TEMP/tools/scripts/report-needs-decision.sh" "$report" >/dev/null || drc=$? + case $drc in + 0) why+=("Claude's report asks for a decision") ;; + 1) ;; + *) why+=("Claude's report could not be read") ;; + esac + + if [ "${#why[@]}" -eq 0 ]; then + backup="backup/${DOWNSTREAM//\//-}-pre-rebase-$(date -u +%Y%m%d)-${OLD:0:12}" + git push "$origin" "$OLD:refs/heads/$backup" + # The lease pins the tip this rebase started from: if anything + # landed on the tree since, the push fails and nothing is lost. + if git push --force-with-lease="refs/heads/$DOWNSTREAM:$OLD" "$origin" \ + "$RESULT:refs/heads/$DOWNSTREAM" 2>"$RUNNER_TEMP/push.err"; then + cat "$RUNNER_TEMP/push.err" >&2 + now=$(git ls-remote "$origin" "refs/heads/$DOWNSTREAM" | cut -f1) + [ "$now" = "$RESULT" ] || { echo "::error::$DOWNSTREAM is $now after the push, expected $RESULT"; exit 1; } + close_superseded "the clean rebase pushed to \`$DOWNSTREAM\` as \`${RESULT:0:12}\`" + { + echo "### Pushed \`$DOWNSTREAM\` → \`${RESULT:0:12}\`" + echo "Was \`${OLD:0:12}\`, kept as \`$backup\`." + echo + cat "$verify" + echo + cat "$report" + } | summary + exit 0 + fi + # Refused: a branch rule, or the tree moved under the lease. A + # clean result is still worth a reviewer's approval, so propose + # it rather than drop it; the land workflow re-checks the tip. + cat "$RUNNER_TEMP/push.err" >&2 + reason=$(grep -m1 -E 'GH0[0-9]+|\[(remote )?rejected\]|error:' "$RUNNER_TEMP/push.err" | + sed -E 's/^remote: *//; s/^[[:space:]]+//; s/[[:space:]]+$//' || true) + why+=("the push to \`$DOWNSTREAM\` was refused${reason:+ (\`$reason\`)}") + git push "$origin" --delete "$backup" || true + fi + + # Not pushed: propose it. + git push "$origin" "$RESULT:refs/heads/$CANDIDATE" + because=$(IFS=';'; echo "${why[*]}") + because=${because//;/ and } + body=$RUNNER_TEMP/pr.md + { + echo "Nightly rebase of \`$DOWNSTREAM\` (\`${OLD:0:12}\`) onto \`$UPSTREAM_BRANCH\` (\`${UP:0:12}\`), driven by Claude. **Not pushed automatically because ${because}.** [Run log]($RUN_URL)." + echo + echo "> [!IMPORTANT]" + echo "> Do not use the merge button: this branch replaces \`$DOWNSTREAM\` rather than merging into it. **Approve** the pull request to land it; the land workflow force-pushes \`$DOWNSTREAM\` to the approved head, pinned to \`${OLD:0:12}\`, and keeps a backup." + echo + echo "## Independent checks" + echo + echo "Builds: **${BUILD}** (x86_64 and arm64 vmlinux with the Edera features built in; per-build results in the run)." + echo + if [ -s "$verify" ]; then cat "$verify"; else echo "The checker did not run."; fi + echo + echo "# Claude's report" + echo + cat "$report" + } >"$body" + if [ "$(wc -c <"$body")" -gt 60000 ]; then + head -c 59000 "$body" >"$body.cut" + printf '\n\n_Truncated; the full report is in the run artifacts._\n' >>"$body.cut" + mv "$body.cut" "$body" + fi + url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$DOWNSTREAM" --head "$CANDIDATE" \ + --title "Rebase $DOWNSTREAM onto $UPSTREAM_BRANCH @ ${UP:0:12}" --body-file "$body") + close_superseded "$url" + echo "### Proposed: $url" | summary