CI's guest suite pulls its Debian image from ToyOSOrg's GHCR copy, with the run's token, by the same digest

[Japabu]

CI's guest suite (guest.yml, called by ci.yml's guest and nightly's tcg) pulls ghcr.io/toyosorg/debian@sha256:a2aa4626…, ToyOSOrg's copy of Docker Hub's debian:sid-20260918, with the run's own GITHUB_TOKEN. The digest is unchanged, so the pin and its provenance are what they were; only the registry and its authentication move. nightly's portability-linux stays on ECR Public's moving debian:sid.

Why

The merge queue's guest / suite for #892 (run 38340843761) failed before any test: public ECR refused the container pull with toomanyrequests: Data limit exceeded. AWS's ECR Public quotas: "For unauthenticated customers, Amazon ECR Public supports up to 500GB of data per month. This is the max amount of data supported and isn't adjustable", plus a 1 pull/s unauthenticated rate (excerpt, from https://docs.aws.amazon.com/AmazonECR/latest/public/public-service-quotas.html). GitHub-hosted runners share their egress, so an anonymous pull from them spends a quota others spend too. Docker Hub was left for ECR in 3fb782c for the same reason, although GitHub's workflow-syntax page says hosted runners are exempt from Docker Hub's limits (excerpt); that exemption did not hold then, so it is not relied on now.

Decisions

• GHCR under ToyOSOrg, filled once. A pull authenticated with GITHUB_TOKEN is the path GitHub documents for Actions: jobs.<job_id>.container.credentials with github.actor and the token, its own example being a ghcr.io image (excerpt, https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idcontainercredentials); "Data transfer is also free when an action installs a package", and GitHub Packages' billing counts a download as Actions' when it logs in with GITHUB_TOKEN (excerpt, https://docs.github.com/en/billing/concepts/product-billing/github-packages). GitHub documents no pull-rate limit for it. Authenticating the ECR pull instead would need an AWS account and an OIDC role; Docker Hub's would need an account and a stored secret.
• Same digest, every byte. The copy took all 14 manifests of the OCI index (seven platforms and their attestations) with skopeo copy --all --preserve-digests; the index read back from GHCR hashes to a2aa4626… by tag and by digest (fill log, the index). The package is linked to ToyOSOrg/ToyOS, which the publishing workflow's repository is granted automatically.
• The package stays private (the orchestrator's ruling, which replaced an earlier ruling to make it public): GitHub's API cannot change a package's visibility, and the pull is authenticated either way. See the last paragraph for the cost.
• packages: read at the job, not the workflow. ci.yml's guest and nightly's tcg grant it to the reusable guest.yml (a called workflow cannot exceed its caller's token); every other job keeps contents: read.
• nightly's portability-linux is untouched. Its premise is a fresh sid machine, image and archive both; a base copied once drifts from sid because deps does no full upgrade, and the job never hit ECR's limit (runs 38040586026, 38128536418, 38233410799 passed on it).
• skopeo is declared in issues/the-build-runs-host-tools-outside-rust-and-qemu.md, the record Arrivals names: Go, so refused; exit, a Rust tool makes the copy or the container goes with the image row. The image row names the GHCR copy for the guest suite and ECR's anonymous sid for portability-linux.
• The fill is not in the tree. It ran once from a temporary fill-ci-image.yml (kept here), dropped from the branch before review; the image is never refilled, and a new pin is a new copy made the same way, and the copy it replaces is deleted.

Gates (head b572aac)

• cargo run -- --ci host: EXIT=0, "Host: 76 step(s), all green" (log).
• The pull, measured at 296a2d5, whose guest.yml and ci.yml this head leaves as they were. Run 38344457897, on 296a2d5 plus the temporary workflow, concluded success: toolchain / build then the real guest.yml with kvm: true, whose container step logged into ghcr.io, pulled Digest: sha256:a2aa4626…, and ran the suite to "Guest: 5 step(s), all green" (log, toolchain).
• Negative control. While the package is private, an anonymous token request for toyosorg/debian:pull answers UNAUTHORIZED (log), so the credentials lines are what make that pull work.
• No guest test is added or changed; the guest suite above is the whole of what it reaches.
• Evidence scrubbed. host-gate.log and docs/github-packages-excerpts.txt carried host paths; both are masked in evidence commit e7f4627aca124e99a3652aafa84dd64924058bd8. The branch is append-only, so 689af7a and 00c678c still hold the unmasked versions.

Unsure

• Whether GHCR throttles token-authenticated pulls at a volume CI reaches; GitHub documents none, and one run is no proof of absence.
• nightly's tcg has not run with its new permissions: block; it is the same block as ci.yml's guest, which did.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

Why the package stays private (orchestrator's ruling, replacing "make it public"). GitHub's API cannot change a package's visibility, and the browser step is the owner's. The pull is authenticated either way. The cost is storage against the organisation's private Packages allowance: about 368 MB for the one copy, which is all seven platforms plus attestations, summed from Docker Hub's tag API for sid-20260918. A new pin is a new copy, so an old copy is deleted when its pin is replaced.

[Japabu]

Review of #903 at 296a2d597 against .claude/agents/reviewer.md on origin/main (1b080a3f1). Net: 4 files changed, 26 insertions(+), 8 deletions(-), all workflow and record. No production code and no tests.

Checked and holding:

• Digest. Both image: lines pin a2aa4626…, unchanged. In the fill log, the GHCR index read back by tag and by digest hashes to a2aa4626…. Docker Hub's tag API maps sid-20260918 to the same digest. In the guest log, docker reports Digest: sha256:a2aa4626….
• Where packages: read is granted. Only at the job level: ci.yml guest, nightly tcg (both callers of guest.yml) and nightly portability-linux. Each of those blocks restates contents: read. The workflow-level permission stays contents: read, and release is still the only job with contents: write.
• Run 38344457897. It ran on push at 7f0ac7815, which is 296a2d597 plus fill-ci-image.yml alone (git diff --stat shows one file). All four jobs succeeded. In guest / suite, the token had Packages: read, the job logged in to ghcr.io, pulled the digest, and ended with "52 passed" and "Guest: 5 step(s), all green".
• Unauthenticated control. An anonymous token request for toyosorg/debian:pull gets UNAUTHORIZED, and the API reports the package as private, linked to ToyOSOrg/ToyOS.
• Cited documents. The excerpts at 00c678cae contain the quoted AWS 500GB/month and 1 pull/s lines, GitHub's container.credentials example, and the billing lines on GITHUB_TOKEN downloads.
• Jobs that must run without the token. None pulls the image. On pull_request, guest / suite is skipped. No host suite, build step or local path names the image (git grep for ghcr, ecr and a2aa4626 finds only the three workflow lines and the issue row). The private package breaks no job.

BLOCKER

• .github/workflows/nightly.yml:88-100 — portability-linux moves from debian:sid as it currently stands to the 2026-09-18 base, which will never be refilled. That breaks what the job is for, and the change has no measured need.
  • What the job was set up to do. It says it builds "from a fresh machine". 3fb782c kept it on "sid as it stands, unpinned" on purpose.
  • The new base drifts. deps runs apt-get install without a full upgrade, so every package outside the install's dependencies stays at 2026-09-18. In run 38344457897, libc6 went 2.43-5 to 2.43-7 only because build-essential pulled it.
  • It can pass falsely. Anything sid drops from its base image after 2026-09-18 stays in this one for good. The job can then pass where a fresh sid machine fails, and the gap only widens as the base ages.
  • The move fixes nothing that was broken. The ECR rate limit hit the merge queue's guest / suite. Nightly's portability-linux passed on ECR's moving tag in runs 38040586026, 38128536418 and 38233410799.
  • --build-only was not measured. On the new image, only deps ran.
  • Fix: revert this hunk and the portability-linux half of the issue row (net about −8 lines). If the owner rules that the job moves anyway, that ruling and a gh workflow run nightly.yml --ref wt/toyos-ci-image with portability-linux and tcg green are what close this. That dispatch would also be the first run of tcg's new permissions: block.

NOTE

• GHCR toyosorg/debian — the package is private on a free-plan org, so its storage counts against GitHub Packages' private allowance. The PR cites only transfer billing. The copy is about 368 MB: the seven platforms plus attestations, summed from Docker Hub's tag API for sid-20260918. CI pulls only amd64, about 50 MB. The PR's "a new pin is a new copy made the same way" would add another full copy. Make the package public (the token pull is unchanged), or state why it is private and what it costs.
• issues/the-build-runs-host-tools-outside-rust-and-qemu.md:27 — the row says the image was "copied once". It does not name the tool that made the copy and will make the next one: skopeo, which is Go, run from a run: step. Arrivals asks for that tool to be declared, with its verdict, in this row.
• ToyOSOrg/evidence 00c678cae — this public repo carries path segments that identify the owner's machine, against root CLAUDE.md. docs/github-packages-excerpts.txt has the unredacted path /Users/jan/.claude/jobs/…. host-gate.log has /Users/<user>/Dev/jan/toyos-ci-image/…, where the username is redacted but the directory names still include it. Scrub both.

SEND BACK

[Japabu]

Rulings on round 1 (#903 (comment)): revert nightly's portability-linux hunk and its issue half, so that job stays on moving sid. Make the GHCR package public: the token pull is unchanged, and no private storage is spent. Declare skopeo in the issue row under Arrivals with its verdict. Scrub the two evidence files in a new commit.

[Japabu]

Round 2 review of #903 at 6f4c227bc against .claude/agents/reviewer.md on origin/main (1b080a3f1). Since 296a2d597: 2 files changed, 5 insertions(+), 10 deletions(-). Whole branch: 4 files changed, 18 insertions(+), 5 deletions(-), all workflow and record. No production code and no tests. The merge base is 858da4d42. Nothing main has added since then touches these four files.

Round 1:

• BLOCKER .github/workflows/nightly.yml portability-linux — CLOSED. git diff origin/main...6f4c227bc -- .github/workflows/nightly.yml now shows only tcg's permissions: hunk, so portability-linux reads byte for byte as on the base: public.ecr.aws/docker/library/debian:sid, no credentials, no packages: read. The issue row's portability-linux half is back to "pulled anonymously from ECR Public's copy of Docker Hub's … sid as it stands".
• NOTE package storage — CLOSED by the replaced ruling. The package stays private. The body's last paragraph gives the reason (no API sets visibility, ghcr-visibility-api.log: PATCH 404, visibility private) and the cost (about 368 MB against the private allowance).
• NOTE skopeo undeclared — CLOSED. It is now a row in issues/the-build-runs-host-tools-outside-rust-and-qemu.md, the file Arrivals names. The row records it as refused (Go), its use as once from a run: step outside main, and its exit as a Rust tool or the container leaving.
• NOTE evidence host paths — CLOSED. At evidence e7f4627ac, neither host-gate.log nor docs/github-packages-excerpts.txt matches /Users/, Dev/, ci-image, jan or .claude. The same holds for host-gate-6f4c227bc.log at afa06f0f6, where paths read <path>/<tmp>. Following the ruling ("in a new commit"), 689af7a and 00c678c keep the earlier text, and the body says so.

Gate at this head: host-gate-6f4c227bc.log ends with "Host: 76 step(s), all green" and EXIT=0. The pull was measured at 296a2d5 (run 38344457897). That still stands for this head: guest.yml and ci.yml are unchanged since, and tcg's permissions: block is identical to ci.yml guest's, which ran.

BLOCKER
(none)

NOTE

• issues/the-build-runs-host-tools-outside-rust-and-qemu.md:30 — this row says "a new pin is a new copy made the same way". The rule that the replaced pin's copy is deleted appears only in the PR body. It needs to be in the record too — why: the package is private, a second copy of about 368 MB doubles the charge to the private allowance, and whoever re-pins reads this row and guest.yml's comment, not a merged PR's body. Fix: add "and the copy it replaces is deleted" to the row's use column.
• PR body, Decisions, "The package is to be public (owner's ruling) … Not yet done" — this is false of the record now. The ruling was replaced, and the body's last paragraph says the package stays private. The bullet contradicts that paragraph and credits the owner with a ruling that has since been replaced.

LAND AFTER NAMED CHANGES