[HWORKS-2991] Document the persistent build daemon, caching and layer reuse - #625
Draft
jimdowling wants to merge 3 commits into
Draft
[HWORKS-2991] Document the persistent build daemon, caching and layer reuse#625jimdowling wants to merge 3 commits into
jimdowling wants to merge 3 commits into
Conversation
… reuse https://hopsworks.atlassian.net/browse/HWORKS-2991 Nothing described here was documented, and all of it is off or conservative by default, so a cluster gets none of it without a deliberate change. A new administrator page covers the persistent BuildKit daemon and how to size its state volume, why client certificates are load-bearing rather than defence in depth, the package cache scopes, dependency locking, and what the cache-key tag protects. It also says where to read build timings, since a slow build is either compilation, transfer or orchestration and those have different fixes. The custom commands guide gains the two directives a build can declare: HOPSWORKS_BUILD_CACHE for toolchain caches, which speed up the work inside a step that still re-runs, and HOPSWORKS_BUILD_HERMETIC for reusing the layer outright, which is framed as the assertion it is. The existing secret reference syntax is written down for the first time as well. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
https://hopsworks.atlassian.net/browse/HWORKS-2991 Records what an environment build actually fetches, which is the question an air-gapped installation has to answer before enabling any of this. The build path adds no new outbound dependency: uv is copied into the base image from a digest-pinned source, packages come from the configured indexes, and the only additional image is the BuildKit daemon, which already has an override for mirroring. States the minimum BuildKit version and why it is higher than it looks: the advisories that matter most are reachable through an ordinary user action, installing a library from a Git URL, and a shared daemon holds state for every project rather than one build. Says plainly that builds produce no SBOM or provenance attestations and that images are not signed, and points image policy at the registry and the output digest the environment history already records, rather than implying the build does work it does not do. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
https://hopsworks.atlassian.net/browse/HWORKS-2991 The chart now ships v0.31.2, so the guidance changes from "pin at least this" to "do not go below this", and repinning gets an explicit warning: buildkitd.gc.keySyntax has to move with the version. A modern daemon accepts the old keepBytes key and reinterprets it as a floor rather than a ceiling, so an operator who changes only the tag gets a cache budget that no longer bounds anything and a state volume that fills. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the environment-build performance work. Backend: logicalclocks/hopsworks-ee#3190. Chart: logicalclocks/hopsworks-helm#2163.
https://hopsworks.atlassian.net/browse/HWORKS-2991
None of this was documented, and all of it is off or conservative by default, so a cluster gets none of it without a deliberate change.
New: Python Build Performance (admin)
setup_installation/admin/build_performance.md, linked under Cluster Configuration.totalKeepBytesbelow the unpacked size of the base images in use makes the daemon slower than no daemon.RUNsteps host networking, so a custom-command script reaches the daemon on127.0.0.1whatever the pod network rules say. Also states plainly that this authenticates but does not authorize.sharedis scoped by index configuration rather than one cluster-wide id.Extended: Custom Commands (user)
HOPSWORKS_BUILD_CACHEwith the full table of supported toolchains, framed as speeding up the work inside a step that still re-runs.HOPSWORKS_BUILD_HERMETIC, framed as the assertion it is, with a note that it does nothing unless the operator has allowed such assertions.NAME=secret:my_secretreference syntax, which already existed and had never been written down.Not done
condawording elsewhere in the python guides still assumes a conda environment. That is accurate for current base images and becomes wrong when the uv virtualenv base image ships (logicalclocks/docker-images#904); it is deliberately left for that change rather than pre-empted here.🤖 Generated with Claude Code