Repository navigation
fix: Don't stack tmpfs mounts on repeated NodePublishVolume - #770
Merged
Merged
Conversation
lfrancke
marked this pull request as draft
November 6, 2026 10:01
Member
Author
|
I ran all kuttl tests and they pass:
|
The kubelet calls NodePublishVolume again for volumes it already published, for example after a kubelet restart, when it reconstructs mounted volumes as uncertain and retries the mount. In privileged mode every call mounted a new tmpfs over the target path. Each extra layer hides the files below it from anything that opens the path later, and costs NodeUnpublishVolume one failed attempt, because it unmounts once and then hits EBUSY removing the directory. The CSI spec requires NodePublishVolume to be idempotent. Reusing the mount means a repeated publish rewrites the existing files, so they are now opened with truncate. Without it, a regenerated key or certificate shorter than the previous one kept the old trailing bytes. This already affected unprivileged mode, which never mounts a tmpfs. A repeated publish still regenerates the secret data, and a running container sees the files rewritten in place.
lfrancke
force-pushed
the
fix/no-stacked-tmpfs-mounts
branch
from
November 6, 2026 10:28
c08a5bb to
089cdba
Compare
Member
Author
|
I also added a test (using jig) for the new behavior here and I could backport it to kuttl but for that to work I'd need to make other changes to the operator so I decided against it. 11/11 passed in 184.8 s.
|
lfrancke
marked this pull request as ready for review
November 6, 2026 15:10
12 tasks
Techassi
self-requested a review
November 9, 2026 08:45
Techassi
requested changes
Nov 9, 2026
Techassi
left a comment
Member
There was a problem hiding this comment.
Only a single small suggestion, otherwise looking good.
Co-authored-by: Techassi <git@techassi.dev>
lfrancke
enabled auto-merge
November 9, 2026 10:26
Techassi
approved these changes
Nov 9, 2026
5 tasks
Member
Author
Release notePreviously, when a secret volume was published again, for example after a kubelet restart, the secret-operator mounted a new tmpfs on top of the existing one each time, which kept Pods in Terminating for minutes. With this release, the existing mount is reused. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Found this while investigating something entirely different.
Basically: secret-operator currently creates a new mount everytime
NodePublishVolumeis called. They stack on top of each other (I had 21 here locally for one of my test clusters :) ) and only a single one is unmounted per retry and then we need to wait for a timeout so it took minutes deleting that.And then Claude found another issue that was uncovered after fixing the first: Because so far we'd get a new mount every time it wasn't a problem but now we
writethe cert to a file which might already exist but we don't truncate it so it could still contain old trailing bytes. That was an easy fix.The CSI spec requires NodePublishVolume to be idempotent and this hould help wtih that.
Author
Reviewer
Acceptance
type/deprecationlabel & add to the deprecation scheduletype/experimentallabel & add to the experimental features tracker