Skip to content

fix: Don't stack tmpfs mounts on repeated NodePublishVolume - #770

Merged
lfrancke merged 2 commits into
mainfrom
fix/no-stacked-tmpfs-mounts
Nov 9, 2026
Merged

lfrancke merged 2 commits into
mainfrom
fix/no-stacked-tmpfs-mounts

Conversation

@lfrancke

@lfrancke lfrancke commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Description

Found this while investigating something entirely different.

Basically: secret-operator currently creates a new mount everytime NodePublishVolume is called. They stack on top of each other (I had 21 here locally for one of my test clusters :) ) and only a single one is unmounted per retry and then we need to wait for a timeout so it took minutes deleting that.

And then Claude found another issue that was uncovered after fixing the first: Because so far we'd get a new mount every time it wasn't a problem but now we write the cert to a file which might already exist but we don't truncate it so it could still contain old trailing bytes. That was an easy fix.

The CSI spec requires NodePublishVolume to be idempotent and this hould help wtih that.

Author

  • Integration tests passed (for non trivial changes)
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

@lfrancke
lfrancke marked this pull request as draft November 6, 2026 10:01
@lfrancke

lfrancke commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

I ran all kuttl tests and they pass:

Test Result Duration
kerberos (krb5 1.21.1) ✅ 90.4 s
tls, 2048 bit, default secret names ✅ 86.6 s
cert-manager-tls ✅ 59.4 s
non-sensitive-data ✅ 43.7 s
listener ✅ 29.6 s
tls, 3072 bit, custom secret names ✅ 19.1 s
tls, 3072 bit, default secret names ✅ 17.0 s
tls, 2048 bit, custom secret names ✅ 13.6 s
tls-truststore, Secret ✅ 8.4 s
tls-truststore, ConfigMap ✅ 8.1 s

The kubelet calls NodePublishVolume again for volumes it already published,
for example after a kubelet restart, when it reconstructs mounted volumes as
uncertain and retries the mount. In privileged mode every call mounted a new
tmpfs over the target path. Each extra layer hides the files below it from
anything that opens the path later, and costs NodeUnpublishVolume one failed
attempt, because it unmounts once and then hits EBUSY removing the directory.
The CSI spec requires NodePublishVolume to be idempotent.

Reusing the mount means a repeated publish rewrites the existing files, so
they are now opened with truncate. Without it, a regenerated key or
certificate shorter than the previous one kept the old trailing bytes. This
already affected unprivileged mode, which never mounts a tmpfs.

A repeated publish still regenerates the secret data, and a running container
sees the files rewritten in place.
@lfrancke
lfrancke force-pushed the fix/no-stacked-tmpfs-mounts branch from c08a5bb to 089cdba Compare November 6, 2026 10:28
@lfrancke

lfrancke commented Nov 6, 2026 •

Copy link
Copy Markdown
Member Author

I also added a test (using jig) for the new behavior here and I could backport it to kuttl but for that to work I'd need to make other changes to the operator so I decided against it.

11/11 passed in 184.8 s.

Test Result Duration
cert-manager-tls ✅ passed 10.4 s
kerberos ✅ passed 15.1 s
listener ✅ passed 10.4 s
non-sensitive-data ✅ passed 5.3 s
republish ✅ passed 21.2 s
tls (custom_secret_names=true, rsa_key_length=2048) ✅ passed 5.4 s
tls (custom_secret_names=false, rsa_key_length=2048) ✅ passed 5.5 s
tls (custom_secret_names=true, rsa_key_length=3072) ✅ passed 10.4 s
tls (custom_secret_names=false, rsa_key_length=3072) ✅ passed 15.4 s
tls-truststore (truststore_target_kind=ConfigMap) ✅ passed 10.4 s
tls-truststore (truststore_target_kind=Secret) ✅ passed 10.4 s

@lfrancke
lfrancke marked this pull request as ready for review November 6, 2026 15:10
@lfrancke lfrancke self-assigned this Nov 6, 2026
@lfrancke lfrancke moved this to Development: Waiting for Review in Stackable Engineering Nov 6, 2026
@Techassi
Techassi self-requested a review November 9, 2026 08:45

@Techassi Techassi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a single small suggestion, otherwise looking good.

@Techassi Techassi moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Nov 9, 2026
Co-authored-by: Techassi <git@techassi.dev>
@lfrancke
lfrancke enabled auto-merge November 9, 2026 10:26
@lfrancke
lfrancke requested a review from Techassi November 9, 2026 10:27
@lfrancke
lfrancke added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 4da9774 Nov 9, 2026
13 checks passed
@lfrancke
lfrancke deleted the fix/no-stacked-tmpfs-mounts branch November 9, 2026 10:46
@lfrancke lfrancke moved this from Development: In Review to Development: Done in Stackable Engineering Nov 9, 2026
@lfrancke

Copy link
Copy Markdown
Member Author

Release note

Previously, when a secret volume was published again, for example after a kubelet restart, the secret-operator mounted a new tmpfs on top of the existing one each time, which kept Pods in Terminating for minutes. With this release, the existing mount is reused.

@lfrancke lfrancke added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Nov 12, 2026
@lfrancke lfrancke moved this from Development: Done to Done in Stackable Engineering Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/26.11.0 release-note Denotes a PR that will be considered when it comes time to generate release notes.

Projects

Development

Successfully merging this pull request may close these issues.

2 participants