Repository navigation
AppVeyor: fix cmd.exe-eaten "||", keep the ccache cache from getting lost, no cache saves from PRs - #3790
Conversation
cmd.exe ignores single quotes, so "||" inside bash -lc '...' is cmd's operator: bash got only the left side, the rest ran in cmd or was silently dropped. - 05/07/08: use "if ! a ; then b ; fi" instead of "||"; note the pitfall in appveyor.yml - 07: dump test logs in the same bash call, keep the real exit code (e.g. 124 from timeout), run make check with -j 2 - 08: drop dead symlink/find line; 7z -mx=3 -mmt=on (default level took ~3.8 min) - 04, new 12-on_success.ps1: run ccache -c before the cache save; ccache overshoots max_size (~120%), and an entry over AppVeyor's per-item limit is not saved, so the next build starts cold - 03: zero ccache stats so reports count this build only - 12-on_success.ps1: also post per-build ccache stats as a job message (shown in the GitHub notification of passed builds) - init: skip cache save in PR builds (one cache per project, each PR build evicted master objects and re-uploaded ~1 GB); save on error only for master pushes Assumes networkupstools#3786 lands with the suggested 02-install.bat line. Refs networkupstools#3786 Signed-off-by: nmbro <nmbro@users.noreply.github.com>
|
A ZIP file with standard source tarball and another tarball with pre-built docs for commit d54560c is temporarily available: NUT-tarballs-PR-3790.zip. |
|
✅ Build nut 2.8.5.5827-master completed (commit d7fe499287 by @nmbro)
|
Done, thanks. I wonder if this also saves some time that can be given to PR job execution, so it is less limited (or not constrained at all) - meaning that 51-minute cut-off introduced recently to ensure that caches are saved consistently. I guess PRs can now use the full hour? I also see many of those timed-out jobs clocking 51 minutes and a few seconds, so cache-saving on top of the scripted 51 minutes take under a minute. Maybe that allowance (now only on branch builds) can be bumped to 58 minutes or so? WDYT? UPDATE: These proposals may go into a future PR, I am eager to merge the currently outstanding recipe fixes - so all other waiting PRs can benefit from the stable baseline. |
…upstools#3786, networkupstools#3790] Signed-off-by: nmbro <nmbro@users.noreply.github.com> Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com> Co-authored-by: Nicolai Brogaard <nmb@minserver.dk>
|
I will make a new branch for the timeout changes so that you can merge this one as is |
PR builds do not save the build cache any more, so they need no early deadline to leave time for it: let them use the whole hour (the step scripts fall back to 3600 seconds when CI_DEADLINE_EPOCH is unset). For other builds, move the deadline from 51 to 57 minutes. Job logs show what still runs after it: 7z of the bundle ~45s (at -mx=3), artifact upload and ccache stats ~10s, cache save 25-40s; deadline-hit master jobs finished about 35s after it. That is ~1.5 minutes, so 57 leaves a margin of about 1.5 minutes before AppVeyor's 60-minute kill. Suggested in review of PR networkupstools#3790. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
PR builds do not save the build cache any more, so they need no early deadline to leave time for it: let them use the whole hour (the step scripts fall back to 3600 seconds when CI_DEADLINE_EPOCH is unset). For other builds, move the deadline from 51 to 57 minutes. Job logs show what still runs after it: 7z of the bundle ~45s (at -mx=3), artifact upload and ccache stats ~10s, cache save 25-40s; deadline-hit master jobs finished about 35s after it. That is ~1.5 minutes, so 57 leaves a margin of about 1.5 minutes before AppVeyor's 60-minute kill. Suggested in review of PR networkupstools#3790. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
Problem
In the AppVeyor step scripts,
cmd.exeonly treats double quotes asquoting, not the single quotes around
bash -lc '...'. So an unquoted||(or|,&&,<,>) in there is cmd's own operator:I reproduced this with a minimal
.bat. On the02-install.batccacheline it means that
ccache -o max_size/compression/sloppinessneverran: the job logs show
max_size5.0 GiB (the default) in 65 of the 70jobs I sampled. The only exceptions were 5 builds from #3784, which set
CCACHE_MAXSIZEin the environment.That is what kept resetting the cache. Over 70 job logs (pushes to
master and PR builds, 2026-10-15 to 2026-10-20):
PACKAGE_VERSIONinconfig.h, see build: keep full git version out of config.h (ccache-friendly) #3759) and kept growing withPR builds, past 1 GB.
limit is 1,048,576,000 bytes compressed). That happened 6 times.
The old entry is gone too, so each time the next build started with
no ccache at all (4% hits, ~160 MB), and the cycle restarted.
04-build_scriptstep: ~15-20 min with a warm cache inPR builds, ~33-37 min cold. Cold jobs take ~50-54 min of the 60-min
limit, and 8 failed jobs stopped at ~51 min, which is the CI deadline.
#3786 fixes the
02-install.batline. This PR contains the same lineas my suggestion there
(#3786 (comment)).
If #3786 lands with that suggestion, the line drops out of this diff on
rebase (a simulated 3-way merge gave no conflicts). Otherwise, I'll
rebase and keep this version.
Changes
02-install.bat(= the Fix build/test errors after recent merges #3786 suggestion),05-after_build.bat,07-test_script.bat,08-after_test.bat: replacea || bwithif ! a ; then b ; fi. The pitfall is documented inappveyor.yml.A static check finds no
|,||,&,&&,<or>outsidedouble quotes in the
bash -lclines any more.ccache can overshoot
max_sizebetween its automatic cleanups (I saw~120%), which is too close to the per-item limit.
04-build_script.batrunsccache -cwhatever the build result,and keeps the build's exit code. With
APPVEYOR_SAVE_CACHE_ON_ERROR,the cache of a failed build is saved right after the failing step,
before
on_failure.on_successstep12-on_success.ps1runs it again forpassed builds, after the tests and the bundle have also compiled.
03-before_build.bat:ccache -z, so the statistics count this buildonly. They were cumulative over the cache lifetime, so the hit rates
in the logs were misleading.
12-on_success.ps1posts this build's ccache hits and cache size asan AppVeyor job message. The GitHub notification template already
includes job messages for passed builds. It uses the invariant
culture, and it is best-effort (a failure there does not fail the
build). Format:
ccache: <hits> of <total> compilations were cache hits (<pct>%); cache size <n> of <max> MiB after cleanupappveyor.ymlinit:APPVEYOR_CACHE_SKIP_SAVE, so they only restore thecache. There is one cache per project, and each save replaces it:
43 of the 46 sampled PR builds saved it, each one evicting master's
objects and uploading ~0.7-1 GB again.
APPVEYOR_SAVE_CACHE_ON_ERRORmoves fromenvironment:toinit,for master pushes only: a failed master build still saves a
partly warmed-up cache, and other branch builds save only when they
pass.
also used by e.g. qreal, EventGhost and simde.
07-test_script.bat:timeout) instead of 1;make checkwith-j 2, the AppVeyor VM size.08-after_test.bat:||problem, only ever removeda non-existent symlink. Had it worked, the symlink would also have
matched
NUT*for 7z and duplicated the archive contents.7z -mx=3 -mmt=on; the default leveltook ~3.8 min of the job time.
Suggestion for the maintainers
The
initline is a convenience only, because a PR can changeappveyor.yml. To really keep PR builds (including from forks) fromwriting to the cache that master builds use, please untick "Save build
cache in Pull Request builds" in the AppVeyor project settings. Today,
a PR could plant ccache objects which a later master build "hits" and
links into its artifacts. This setting is not available in YAML.
Timings (local, MSYS2 on an 8-core machine, limited to 2 jobs like the AppVeyor VM)
7z on a 679 MB bundle, two threads:
-mx=5)-mx=3-mx=1make check:-j 2Testing
cmd.exein MSYS2 with a temporaryCCACHE_DIRandCCACHE_MAXSIZE=750MiBas inappveyor.yml:02line: reportsmax_size=750.0 MiB compression=true;03stats reset: "Statistics zeroed";04line: in a dir withoutci_build.sh, the cleanup ran andexit code 127 was kept.
12-on_success.ps1and theinitlogic: tested with stub AppVeyorcmdlets and variables, for a PR build, a master push and another
branch.
appveyor.ymlparses, all files are ASCII, and the.batfiles keepCRLF.
notification.
AI disclosure
Prepared with Claude Code (Claude Opus 5.5, model
claude-opus-5-5):log analysis, changes, local tests and this description. I reviewed
and adjusted the changes (e.g. merged the on_success cleanup into the
PowerShell step), and I take responsibility for the submission.
🤖 Generated with Claude Code