Repository navigation
CI: speed up AppVeyor build, fix ccache misses across PRs - #3754
Conversation
dllldd_with_tools() ran a `find` per DLL name over DESTDIR and over each input directory. Once recursion reaches e.g. /mingw64/bin, this means hundreds of walks over a large directory, which is slow on MSYS2 (about 5 minutes of `make install-win-bundle` in AppVeyor CI). List those locations once per call and match base names with awk. Listing order is kept, so the first hit is the same file as before. Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
Single-threaded LZMA at -mx=6 took 1-2 minutes per run to re-pack the already-compressed ccache entry. Store entries as-is instead, and cap ccache at 750MB to stay below the 1GB build cache limit for PRs along with the other cached entries. Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
|
A ZIP file with standard source tarball and another tarball with pre-built docs for commit 39ffa4a is temporarily available: NUT-tarballs-PR-3754.zip. |
AC_INIT takes the version from gitlog2version.sh, which includes the number of commits in a PR (e.g. 2.8.5.1466.13), and puts it into config.h as PACKAGE_VERSION. With -g3 every object file embeds all macros it saw, so each new commit count meant a full rebuild on a 2-CPU AppVeyor worker (~50 min), and ccache hits only for PRs with a commit count seen before. Add opt-in CI_AUTOGEN_STABLE_VERSION=true to drop that component via NUT_VERSION_FORCED for autogen.sh only, and enable it for AppVeyor. `make` still generates nut_version.h from Git, so programs report the actual version. Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
|
@jimklimov question on This commit forces the version without OK with that, or would you rather use plain |
AppVeyor kills the job at 60 minutes without saving the build cache, so a cold-ccache build never warms up (and an interrupted cache save in build 5650 lost the ccache entry entirely). Set a deadline at 51 minutes after job start in `init`, and run the long bash steps (prereqs, build, check, spellcheck, bundle) under `timeout` with the time left. If they would run past it, the job fails instead, and APPVEYOR_SAVE_CACHE_ON_ERROR keeps what was built, with at least ~5 minutes left for 7z, artifacts and the cache save. Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
The cache misses come from the ccache direct mode hashing config.h contents, not from -g3 (ccache does not treat -g3 specially). Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
|
Correction to my question above: |
|
❌ Build nut 2.8.5.5657-master failed (commit dfebc82854 by @nmbro) |
|
✅ Build nut 2.8.5.5658-master completed (commit d009a938d9 by @nmbro)
|
|
✅ Build nut 2.8.5.5658-master completed (commit d009a938d9 by @nmbro) |
|
✅ Build nut 2.8.5.5659-master completed (commit f894c3a573 by @nmbro)
|
|
Welcome back appveyor! |
When cached certificates failed check_NIT_certs, the cache dir was removed and immediately re-created empty. The cache is populated only if that dir is absent, so it stayed empty forever, and every run (seen on AppVeyor) generated new certificates after "GRAVE PROBLEM" noise. Do not re-create the cache dir there. Also fix a `|| \` typo (vs. `|| \` line continuation), which ran a command named `\` and then the `cp -L` fallback unconditionally. Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
|
✅ Build nut 2.8.5.5660-master completed (commit eac58be52b by @nmbro)
|
`strings` scanned whole files, including debug sections: for the NUT for Windows bundle (616MB of -g3 binaries) that produced 561MB of text, all piped through tr/grep/sort, which dominated the DLL scan of `make install-win-bundle` (~2 min in the first pass on AppVeyor). With `strings -d` (data sections only, as older binutils did by default) it is 2.8MB, and the same candidate DLL names. Fall back to a whole-file scan if `strings` lacks the option. Similarly use `objdump -p` (private headers, with the PE import table and ELF NEEDED entries) instead of `-x`, which also dumped symbol tables and relocations (37MB vs 9.8MB of text here). Checked against the AppVeyor build 5659 bundle: identical results and warnings to the original script. Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
|
✅ Build nut 2.8.5.5661-master completed (commit 245ab3e7ac by @nmbro)
|
|
✅ Build nut 2.8.5.5662-master completed (commit a6d701d65d by @nmbro)
|
Add combase, hid, setupapi, shfolder, wship6 and xmllite to the list of OS-provided DLLs, which were reported as not found (strings matcher) for every NUT for Windows bundle on AppVeyor. The `%s.dll` format-string filter missed a `\%s.dll` hit (backslash path separator in a format string); `read` later stripped the backslash, so it was reported as not found as `%s.dll`. Allow a leading path. Checked against the AppVeyor build 5659 bundle: identical DLL list, these 7 warnings gone, no new ones. Use of coding helper tools and AI disclosed: Claude Code was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
|
@jimklimov could you enable "Rolling builds" in the AppVeyor project settings (UI-only, not possible via |
nmbro
changed the title
|
✅ Build nut 2.8.5.5663-master completed (commit 53afc79436 by @nmbro)
|
jimklimov
added
Windows
CI
jimklimov
added
the
SSL/NSS
|
Fair point, Rolling Builds enabled (only for PRs, not branches like master increments). Thanks for the heads-up! |
jimklimov
left a comment
jimklimov
left a comment
There was a problem hiding this comment.
Great finds, thanks a lot!
I've noted some questions/suggestions which may well be addressed later, as I am eager to reap the benefits of this and other CI PRs as quickly as possible, to speed up unclogging the CI queues :D
| # this setting, so `make` still generates nut_version.h (and the | ||
| # UPS_VERSION reported by programs) from actual Git metadata. | ||
| if [ x"${CI_AUTOGEN_STABLE_VERSION-}" = xtrue ] && [ -z "${NUT_VERSION_FORCED-}" ] ; then | ||
| _NUT_VERSION_STABLE="`NUT_VERSION_QUERY=VER50 ./tools/gitlog2version.sh 2>/dev/null`" || _NUT_VERSION_STABLE="" |
There was a problem hiding this comment.
This is an interesting trick, especially if constrained to autotools but not impacting the ultimate build. I can't even say if I intended to do something like that earlier, when making this tool, and forgot. So kudos! :D
I wonder if it makes sense to go even further and just use NUT_VERSION_QUERY=SEMVER to populate NUT_VERSION_FORCED here, so the same cache data can be used as long as it is relevant after an official release? The fourth component (commits on master since release) tends to get outdated when PRs are merged. Though maybe occasionally rebuilding everything "honestly" is not that bad, if it would fit into the time frame.
As it stands here and now, the cache would be relevant at least for all PRs' iterations based on top of current master branch.
There was a problem hiding this comment.
Another thought, following the comments in PR #3753, with all of these merged - would each PR running "in its own cache scope" start with a blank cache or inherit the master-branch cache as a starting point?
There was a problem hiding this comment.
They inherit from: own PR, base branch, default branch
No re-run reuse in PRs. A PR’s second push won’t reuse its own first build’s ccache. It starts from master’s cache again. That’s the trade-off you’re accepting, and it’s usually fine.
So the PR already does the restore on PRs
cache saves are only explicitly on pushed ie. merges or direct pushes to the default branch
| APPVEYOR_CACHE_ENTRY_ZIP_ARGS: -t7z -mx=0 | ||
| # Keep config.h stable across PRs, so ccache can reuse -g3 objects | ||
| # (see ci_build.sh for details): | ||
| CI_AUTOGEN_STABLE_VERSION: true |
There was a problem hiding this comment.
To clarify in no uncertain words: all this hassle with PACKAGE_VERSION only impacts debug builds, but not ordinary "release" ones? Would those be with explicit -g0?
Is this embedding of all macros, whether used or not, into the binaries, a feature/constraint of specific toolkit (gcc/clang/linker...)? Are there clear options against this activity?
Looking forward, if debug data were to be offloaded into separate files (like many packaging systems do) - is this gonna hit a similar snag by having nominally enabled debug build (just the linker outputs fall into many files not one)?
MAYBE then the fix applied in this file to honour the CI_AUTOGEN_STABLE_VERSION setting belongs actually in autogen.sh file itself, so e.g. package building recipes can use it without calling ci_build.sh (normal packages are likely to have fixed versions from tarballs anyway, but e.g. OBS PR builds might benefit from this). Also, makes sense to mention this in guides how to build NUT, especially repetitively (packagers' guide, developers' notes).
There was a problem hiding this comment.
I'll be posting the doc snippet about this.
There was a problem hiding this comment.
yeah, I think the better approach would be to take the PACKAGE_VERSION out of config.h since that hits way too many files, and maybe have the version number a different place?
There was a problem hiding this comment.
I am not sure that can be done cleanly, part of autotools standard generated code. Like, we can post-process what they make somehow (and probably not in configure.ac payload body), but it would not be a proper approach.
UPDATE: Newly proposed PR #3759 seems to prove this point wrong, thanks!
…tools#3754, networkupstools#3756] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…st optimizations" chapter [networkupstools#3754] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
|
Feel free to merge this when you are ready, we should probably look at a more general stable and ccache friendly version file fix could be that the rest of the CI could benefit from it |
include/config.h is included by every source file, and carried the full git-derived version (e.g. 2.8.5.1466.13) in PACKAGE_VERSION, PACKAGE_STRING and VERSION macros. It changed with every commit, so ccache direct mode missed for all objects on each new commit or fresh CI checkout. Without -g3, preprocessed mode mostly recovered the hits (no code uses these macros), but with -g3 (e.g. AppVeyor) the preprocessed output keeps all #define lines, so every object was recompiled. Pass the base SEMVER (e.g. 2.8.5) to AC_INIT, so config.h only changes with releases, and set the shell PACKAGE_VERSION/PACKAGE_STRING right after it back to the full VER50 value, so all @PACKAGE_VERSION@ substitutions (Makefiles, *.pc files, packaging recipes, tarball names, configure reports) keep the full version. Use AM_INIT_AUTOMAKE no-define (PACKAGE and VERSION macros are unused) and reset VERSION for distdir. Programs still report the full version via NUT_VERSION_MACRO in include/nut_version.h, consumed by two files. Checked with CCACHE_DEBUG=1 builds across two commits (WSL gcc 13, ubuntu-plucky gcc 14): config.h byte-identical, all objects direct hits except upsclient.o and common-nut_version.o. Also checked a dist tarball build (VERSION_DEFAULT), NUT_VERSION_FORCED, and autoconf 2.69 on CentOS 7 and Ubuntu Trusty. Docs go into the "Repetitive build/test optimizations" chapter of INSTALL.nut.adoc. Follow-up to PR networkupstools#3754. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: nmbro <nmbro@users.noreply.github.com>
…CACHE_MAXSIZE envvar if possible [networkupstools#3754] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…CACHE_MAXSIZE envvar if possible [#3754] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
AppVeyor builds kept hitting the 60 min limit. Causes and fixes:
PACKAGE_VERSIONinconfig.hincludes the PR's commit count (e.g.2.8.5.1466.13), and ccache's direct mode hashesconfig.h, so a new count meant a full rebuild (~35-50 min on 2 CPUs). Opt-inCI_AUTOGEN_STABLE_VERSION(on for AppVeyor) drops that component forautogen.shonly;nut_version.h/UPS_VERSIONstill come from git.timeoutwith a deadline at 51 min, so the job fails instead andAPPVEYOR_SAVE_CACHE_ON_ERRORkeeps the cache.install-win-bundle:dllldd.shranstringsover whole files, including debug sections (561MB of text for this bundle);strings -dgives 2.8MB with the same DLL names. Alsoobjdump -pinstead of-x, and a few more system DLLs filtered. Checked against a real AppVeyor bundle: identical DLL list.tests/NIT/nit.sh.Measured per phase (AppVeyor, warm cache):
install-win-bundle)Total job time varies a few minutes between runs with VM load, unrelated to these changes (36-40 min warm); a cold build previously timed out, and now finishes or fails at 51 min with the cache saved.
Open question for maintainers: see the comment about the stable
AC_INITversion.Use of coding helper tools and AI disclosed: Claude Code was used for development assistance.
🤖 Generated with Claude Code