Flake: sbt 1.2.8 warm-cache update in Dockerfile.sbt hangs until the 45-min job timeout #1665

Activity

1. Mikola Lysenko (mikolalysenko)

   added

   ci-janitorOpened by the CI janitor routine (flakes, redundant tests, CI perf)Opened by the CI janitor routine (flakes, redundant tests, CI perf)

   on Oct 22, 2026
2. mikolalysenko commented on Oct 22, 2026

   

   Mikola Lysenko (mikolalysenko)

   on Oct 22, 2026

   CollaboratorAuthor

   More actions

   Two more occurrences, both in the workflow_dispatch CI run 39745825228 on ci-janitor/uv-settings-test-windows (#1663):

   • Attempt 1: e2e-docker (sbt), job 119716408680. Cancelled at the 45-minute limit (08:16:57 to 09:02:02 UTC).
   • Attempt 2: e2e-docker (sbt), job 119735730658. Cancelled the same way (09:02:36 to 09:47:38 UTC). Its last output is [launcher] getting org.scala-sbt sbt 1.2.8 and then Updating ProjectRef(uri("file:/tmp/warm-1.2.8/project/"), "warm-1-2-8-build")... at 09:29:53. Nothing more was logged until the cancel at 09:47:36, an 18-minute silent stall.

   Impact: the dispatch run's ci-ok went red on a test-only PR, and the final-review routine disarmed auto-merge on #1663. For comparison, on main's nightly with a warm image cache this job took about 6 minutes (2026-10-21, 05:52 to 05:59). The stall shows up when the image is built without a cache, as on a dispatch run on a branch.

   ---

   Generated by Claude Code
3. Mikola Lysenko (mikolalysenko)

   added

   agent:triaged

   priority:p3

   on Oct 22, 2026
4. mikolalysenko commented on Oct 22, 2026

   

   Mikola Lysenko (mikolalysenko)

   on Oct 22, 2026

   CollaboratorAuthor

   More actions

   CI janitor (2026-10-22 15:5x UTC): another occurrence, plus evidence that changes the likely cause.

   4th occurrence: main's nightly 39730006970, coverage-docker (sbt) (job 119664378238). The SBT_WARM_VERSIONS=1.2.8 1.13.0 slice sat on Updating ProjectRef(... "warm-1-2-8-build")... from 05:53:28 until the 45-min cancel at 06:38. So it does hit main's nightly too, not just cold dispatch builds. The same night's full-image e2e-docker (sbt) passed, and so did every sbt image build on 10-18 to 10-21 (about 3 min for the build step).

   It is probably not a no-timeout hang. The full-image build in dispatch attempt 2 (job 119735730658) warms 0.13.18 before 1.2.8, and 0.13.18 logs every Ivy Resolving line. The step was running at the normal rate, just very slowly:

   • [launcher] getting org.scala-sbt sbt 0.13.18 took 637 s (09:03:01 to 09:13:38).
   • After that, almost every Resolving <module> advanced in steps of ~20 s, 40 s, 60 s or 80 s. Examples: scala-library 2.10.7 at +40 s, scala-compiler at +40 s, oss-parent;9 at +60 s, codehaus-parent;4 at +80 s. That is a fixed per-request timeout hit once or more per module, so sockets do time out.
   • The jars themselves downloaded in 0.1 to 1.6 s each, from both repo1.maven.org and repo.typesafe.com. Its whole update finished at 1608 s.
   • Then 1.2.8 started at 1613 s. 1.x Ivy doesn't print Resolving lines, so the same slow crawl over its much larger build definition looks like silence. It was cancelled 18 min later.

   So the likely cause is one resolver host in Ivy's chain that times out on metadata lookups. sbt's chain resolver queries every resolver for each module, so one dead host costs ~20 s per module per lookup. It doesn't serve any artifact we use. Both the launcher's repositories and sbt's default build resolvers include repo.scala-sbt.org (scalasbt/maven-releases, ivy-snapshots, sbt-plugin-releases), and every artifact we actually fetched came from the other two hosts. That makes repo.scala-sbt.org the prime suspect, but this is still unconfirmed. I couldn't check it from this session, because the agent proxy blocks the host.

   What that means for the fix:

   • Adding -Dsun.net.client.defaultReadTimeout likely won't help. Requests already time out; there are just ~100 of them in a row.
   • A timeout bound plus one retry on each warm sbt ... update turns a 45-min silent stall into a fast, clear failure. But the stall lasted at least 05:53 to 09:47 today, so a retry minutes later would likely stall again.
   • The fix that addresses the cause is to keep the stalling host out of the warm boots' resolver chain. For example, a repositories file plus -Dsbt.override.build.repos=true limited to local, Maven Central and typesafe-ivy-releases (still needed for 0.13.x). This has to match what the docker_e2e_sbt cells resolve with, or Ivy's resolver-tagged cache entries won't be reused and the cells boot cold. It needs a real docker build on a runner to confirm, which this session can't do.
   • Before choosing, a cheap diagnostic: a dispatch run with -debug on the 1.2.8 warm boot (or -Divy.message.logger.level=4) will name the host that times out.

   No PR from this run: I can't validate a resolver change without a runner-side image build. This isn't merge-blocking (coverage-docker and e2e-docker only run on schedule and dispatch), so it stays a step-B candidate for a run that can confirm the host.

   ---

   Generated by Claude Code
5. mikolalysenko commented on Oct 23, 2026

   

   Mikola Lysenko (mikolalysenko)

   on Oct 23, 2026

   CollaboratorAuthor

   More actions

   [agent] Re-verified (CI lane, 2026-10-23 07:5x UTC). The stall did not recur on the next nightly: run 39907645034 (aa69125) built the sbt image in normal time, with coverage-docker (sbt) at 05:52:56 to 05:57:01 (4 min) and e2e-docker (sbt) at 05:52:56 to 05:58:38 (6 min). Both e2e-extended sbt 1.13.0 legs also passed in 2 to 4 min.

   That fits the transient slow-resolver diagnosis in the comment above: all 4 hangs fall inside 2026-10-22 05:53 to 09:47 UTC. Not claiming a fix this run. A timeout bound on the warm sbt update calls would only shorten a future stall, not make the leg pass. If it shows up again on a nightly, that is the time to pin the build resolvers (~/.sbt/repositories with -Dsbt.override.build.repos=true) so the warm step skips repo.scala-sbt.org.

   ---

   Generated by Claude Code