Skip to content

AppVeyor: fix cmd.exe-eaten "||", keep the ccache cache from getting lost, no cache saves from PRs - #3790

Merged
jimklimov merged 1 commit into
networkupstools:masterfrom
nmbro:appveyor-cmd-quoting
Oct 21, 2026
Merged

jimklimov merged 1 commit into
networkupstools:masterfrom
nmbro:appveyor-cmd-quoting

Conversation

@nmbro

@nmbro nmbro commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Problem

In the AppVeyor step scripts, cmd.exe only treats double quotes as
quoting, not the single quotes around bash -lc '...'. So an unquoted
|| (or |, &&, <, >) in there is cmd's own operator:

  • bash gets only the text before it;
  • cmd runs the rest as a cmd command if bash failed, or skips it.

I reproduced this with a minimal .bat. On the 02-install.bat ccache
line it means that ccache -o max_size/compression/sloppiness never
ran
: the job logs show max_size 5.0 GiB (the default) in 65 of the 70
jobs I sampled. The only exceptions were 5 builds from #3784, which set
CCACHE_MAXSIZE in the environment.

That is what kept resetting the cache. Over 70 job logs (pushes to
master and PR builds, 2026-10-15 to 2026-10-20):

  • ccache grew by ~155 MB per master push (each one changes
    PACKAGE_VERSION in config.h, see build: keep full git version out of config.h (ccache-friendly) #3759) and kept growing with
    PR builds, past 1 GB.
  • AppVeyor then refuses to save it ("Unable to save cache": the per-item
    limit is 1,048,576,000 bytes compressed). That happened 6 times.
    The old entry is gone too, so each time the next build started with
    no ccache at all (4% hits, ~160 MB), and the cycle restarted.
  • Effect on the 04-build_script step: ~15-20 min with a warm cache in
    PR builds, ~33-37 min cold. Cold jobs take ~50-54 min of the 60-min
    limit, and 8 failed jobs stopped at ~51 min, which is the CI deadline.

#3786 fixes the 02-install.bat line. This PR contains the same line
as my suggestion there
(#3786 (comment)).
If #3786 lands with that suggestion, the line drops out of this diff on
rebase (a simulated 3-way merge gave no conflicts). Otherwise, I'll
rebase and keep this version.

Changes

  • 02-install.bat (= the Fix build/test errors after recent merges #3786 suggestion), 05-after_build.bat,
    07-test_script.bat, 08-after_test.bat: replace a || b with
    if ! a ; then b ; fi. The pitfall is documented in appveyor.yml.
    A static check finds no |, ||, &, &&, < or > outside
    double quotes in the bash -lc lines any more.
  • Enforce the ccache size limit before AppVeyor saves the cache.
    ccache can overshoot max_size between its automatic cleanups (I saw
    ~120%), which is too close to the per-item limit.
    • 04-build_script.bat runs ccache -c whatever the build result,
      and keeps the build's exit code. With APPVEYOR_SAVE_CACHE_ON_ERROR,
      the cache of a failed build is saved right after the failing step,
      before on_failure.
    • The new on_success step 12-on_success.ps1 runs it again for
      passed builds, after the tests and the bundle have also compiled.
  • 03-before_build.bat: ccache -z, so the statistics count this build
    only. They were cumulative over the cache lifetime, so the hit rates
    in the logs were misleading.
  • 12-on_success.ps1 posts this build's ccache hits and cache size as
    an AppVeyor job message. The GitHub notification template already
    includes job messages for passed builds. It uses the invariant
    culture, and it is best-effort (a failure there does not fail the
    build). Format:
    ccache: <hits> of <total> compilations were cache hits (<pct>%); cache size <n> of <max> MiB after cleanup
  • appveyor.yml init:
    • PR builds set APPVEYOR_CACHE_SKIP_SAVE, so they only restore the
      cache. There is one cache per project, and each save replaces it:
      43 of the 46 sampled PR builds saved it, each one evicting master's
      objects and uploading ~0.7-1 GB again.
    • APPVEYOR_SAVE_CACHE_ON_ERROR moves from environment: to init,
      for master pushes only: a failed master build still saves a
      partly warmed-up cache, and other branch builds save only when they
      pass.
    • This is the pattern from https://www.appveyor.com/docs/build-cache/,
      also used by e.g. qreal, EventGhost and simde.
  • 07-test_script.bat:
    • dumps the test logs from the same bash call;
    • keeps the real exit code (e.g. 124 from timeout) instead of 1;
    • runs make check with -j 2, the AppVeyor VM size.
  • 08-after_test.bat:
    • drops a line which, due to the same || problem, only ever removed
      a non-existent symlink. Had it worked, the symlink would also have
      matched NUT* for 7z and duplicated the archive contents.
    • compresses the artifact with 7z -mx=3 -mmt=on; the default level
      took ~3.8 min of the job time.

Suggestion for the maintainers

The init line is a convenience only, because a PR can change
appveyor.yml. To really keep PR builds (including from forks) from
writing to the cache that master builds use, please untick "Save build
cache in Pull Request builds" in the AppVeyor project settings. Today,
a PR could plant ccache objects which a later master build "hits" and
links into its artifacts. This setting is not available in YAML.

Timings (local, MSYS2 on an 8-core machine, limited to 2 jobs like the AppVeyor VM)

7z on a 679 MB bundle, two threads:

7z level time size
default (-mx=5) 98 s 52 MB
-mx=3 10 s 76 MB
-mx=1 12 s 199 MB (switches to a weaker method)

make check:

time result
serial 279 s all 16 tests passed
-j 2 184-207 s all 16 tests passed in each of 7 serial/parallel runs

Testing

  • Ran these lines through cmd.exe in MSYS2 with a temporary
    CCACHE_DIR and CCACHE_MAXSIZE=750MiB as in appveyor.yml:
    • the 02 line: reports max_size=750.0 MiB compression=true;
    • the 03 stats reset: "Statistics zeroed";
    • the 04 line: in a dir without ci_build.sh, the cleanup ran and
      exit code 127 was kept.
  • 12-on_success.ps1 and the init logic: tested with stub AppVeyor
    cmdlets and variables, for a PR build, a master push and another
    branch.
  • appveyor.yml parses, all files are ASCII, and the .bat files keep
    CRLF.
  • NEWS bullet: passes the spellcheck.
  • Not tested: an actual AppVeyor run, which this PR provides.
    • A PR build log should show no "Updating build cache..." line.
    • A failed master build should still show one.
    • Passed builds should show the ccache line in the GitHub
      notification.

AI disclosure

Prepared with Claude Code (Claude Opus 5.5, model claude-opus-5-5):
log analysis, changes, local tests and this description. I reviewed
and adjusted the changes (e.g. merged the on_success cleanup into the
PowerShell step), and I take responsibility for the submission.

🤖 Generated with Claude Code

cmd.exe ignores single quotes, so "||" inside bash -lc '...' is
cmd's operator: bash got only the left side, the rest ran in cmd
or was silently dropped.

- 05/07/08: use "if ! a ; then b ; fi" instead of "||"; note the
  pitfall in appveyor.yml
- 07: dump test logs in the same bash call, keep the real exit
  code (e.g. 124 from timeout), run make check with -j 2
- 08: drop dead symlink/find line; 7z -mx=3 -mmt=on (default
  level took ~3.8 min)
- 04, new 12-on_success.ps1: run ccache -c before the cache save;
  ccache overshoots max_size (~120%), and an entry over AppVeyor's
  per-item limit is not saved, so the next build starts cold
- 03: zero ccache stats so reports count this build only
- 12-on_success.ps1: also post per-build ccache stats as a job message
  (shown in the GitHub notification of passed builds)
- init: skip cache save in PR builds (one cache per project, each
  PR build evicted master objects and re-uploaded ~1 GB); save on
  error only for master pushes

Assumes networkupstools#3786 lands with the suggested 02-install.bat line.
Refs networkupstools#3786

Signed-off-by: nmbro <nmbro@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

A ZIP file with standard source tarball and another tarball with pre-built docs for commit d54560c is temporarily available: NUT-tarballs-PR-3790.zip.

@AppVeyorBot

Copy link
Copy Markdown

✅ Build nut 2.8.5.5827-master completed (commit d7fe499287 by @nmbro)

@jimklimov

jimklimov commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

please untick "Save build cache in Pull Request builds" in the AppVeyor project settings

Done, thanks. I wonder if this also saves some time that can be given to PR job execution, so it is less limited (or not constrained at all) - meaning that 51-minute cut-off introduced recently to ensure that caches are saved consistently. I guess PRs can now use the full hour?

I also see many of those timed-out jobs clocking 51 minutes and a few seconds, so cache-saving on top of the scripted 51 minutes take under a minute. Maybe that allowance (now only on branch builds) can be bumped to 58 minutes or so? WDYT?

UPDATE: These proposals may go into a future PR, I am eager to merge the currently outstanding recipe fixes - so all other waiting PRs can benefit from the stable baseline.

@jimklimov jimklimov added Windows CI Entries related to continuous integration infrastructure (here CI = tools + scripts + recipes) AI For good or bad, machine tools are upon us. Humans are still the responsible ones. labels Oct 21, 2026
@jimklimov jimklimov added this to the 2.8.6 milestone Oct 21, 2026
jimklimov added a commit to jimklimov/nut that referenced this pull request Oct 21, 2026
…upstools#3786, networkupstools#3790]


Signed-off-by: nmbro <nmbro@users.noreply.github.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>

Co-authored-by: Nicolai Brogaard <nmb@minserver.dk>
@jimklimov
jimklimov marked this pull request as ready for review October 21, 2026 12:06
@nmbro

nmbro commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

I will make a new branch for the timeout changes so that you can merge this one as is

@jimklimov
jimklimov merged commit ae8da2b into networkupstools:master Oct 21, 2026
29 of 31 checks passed
@jimklimov jimklimov moved this to Done in NUT for Windows Oct 21, 2026
nmbro added a commit to nmbro/nut that referenced this pull request Oct 21, 2026
PR builds do not save the build cache any more, so they need no early
deadline to leave time for it: let them use the whole hour (the step
scripts fall back to 3600 seconds when CI_DEADLINE_EPOCH is unset).

For other builds, move the deadline from 51 to 57 minutes. Job logs
show what still runs after it: 7z of the bundle ~45s (at -mx=3),
artifact upload and ccache stats ~10s, cache save 25-40s; deadline-hit
master jobs finished about 35s after it. That is ~1.5 minutes, so 57
leaves a margin of about 1.5 minutes before AppVeyor's 60-minute kill.

Suggested in review of PR networkupstools#3790.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: nmbro <nmbro@users.noreply.github.com>
nmbro added a commit to nmbro/nut that referenced this pull request Oct 21, 2026
PR builds do not save the build cache any more, so they need no early
deadline to leave time for it: let them use the whole hour (the step
scripts fall back to 3600 seconds when CI_DEADLINE_EPOCH is unset).

For other builds, move the deadline from 51 to 57 minutes. Job logs
show what still runs after it: 7z of the bundle ~45s (at -mx=3),
artifact upload and ccache stats ~10s, cache save 25-40s; deadline-hit
master jobs finished about 35s after it. That is ~1.5 minutes, so 57
leaves a margin of about 1.5 minutes before AppVeyor's 60-minute kill.

Suggested in review of PR networkupstools#3790.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: nmbro <nmbro@users.noreply.github.com>
@nmbro
nmbro deleted the appveyor-cmd-quoting branch October 21, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI For good or bad, machine tools are upon us. Humans are still the responsible ones. CI Entries related to continuous integration infrastructure (here CI = tools + scripts + recipes) Windows

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants