Fix kbuild pahole hang - #3219
Merged
Merged
Conversation
pahole 1.29 deadlocks when its threaded BTF encoder meets a type it cannot represent. The worker that hits the error terminates without signalling its peers, so the main thread and one remaining worker park on futexes and never return. Any -j >= 2 triggers it, provided there are enough compilation units for the pool to run concurrently, and vmlinux always has enough. The result is a build that hangs forever instead of failing. ARC reaches this because it links libgcc, whose libgcc2.c is built with debug info and carries complex float base types in its DWARF. pahole cannot encode those in BTF and says so: Complex, interval and imaginary float types are not supported Error while encoding BTF. Under 1.29 that message is followed by the deadlock. Builds seen in production sat Running for more than six days, each holding a whole node until it was reaped by hand. 1.31 reports the same unsupported-type error and exits cleanly, so the build fails normally. This reproduces without a kernel tree: 1000 trivial translation units, a few containing a _Complex float member, linked with -g. 1.29 exits 0 at -j1 but hangs at -j2 and -j8, while 1.31 exits 0 at every -j. Note this only stops the hang. ARC still produces no BTF and so still fails, later and visibly, in resolve_btfids. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
backoffLimit is a Job-level field, but it was written under spec.template.spec. It is not part of the PodSpec schema there, so the API server drops it and the jobs ran with the default limit of 6: every failing build was retried seven times over, multiplying one bad build into a pile of dead pods. Move it to the Job spec, where the intended limit of 1 takes effect. Also add activeDeadlineSeconds, so a job cannot run indefinitely. A build whose toolchain wedges keeps its pod Running and holds a whole node, and nothing reclaims it: ttlSecondsAfterFinished only applies once a job has finished, and the timeout the API tracks on the node is advisory and is never enforced against the Job. Builds have been observed Running for over six days this way, with six of the nine nodes in one cluster held by them. Default to 21600 seconds, matching the six hour timeout the API already records, and allow a job to override it. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.