Mounted secrets and the restarter.stackable.tech/expires-at can drift apart #772

Affected Stackable version

all

Current and expected behavior

Description

When Kubernetes calls NodePublishVolume again for a volume that is already mounted into a running Pod (for example after a kubelet restart), secret-operator issues a new certificate and moves the Pod's restarter.stackable.tech/expires-at.* annotation to the new certificate's expiry. The running container keeps the old files. commons-operator then schedules the eviction based on the new expiry, so the Pod can keep serving an expired certificate. No eviction is attempted before it expires.

Root cause

node_publish_volume performs the following actions in order:

1. tag_pod patches the annotation before any file is written.
2. prepare_secret_dir warns about an existing target directory (Tried to create volume path that already exists) then mounts a new tmpfs on top of the path.
3. save_secret_data writes the new files into that new tmpfs.

The container's bind mount still refers to the original tmpfs, so it never sees the new files.

On Trino there is a second contributing factor: the start command copies keystore.p12 once into /stackable/server_tls/, so even a refreshed volume would not reach the file Trino reads until the process restarts.

Reproduction

Tested on minikube with secret-operator 0.0.0-dev (privileged CSI driver):

1. Create a Pod with an ephemeral secrets.stackable.tech volume. Use an autoTls SecretClass and tls-pkcs12 format.
2. Record the expires-at.* annotation, the keystore hash inside the container, and the tmpfs mounts on the volume path on the node.
3. Restart the kubelet on the node: systemctl restart kubelet.

Observed after the restart:

• The operator logs a second Received NodePublishVolume and then warns about an existing volume path.
• The node has two tmpfs mounts stacked on the volume path.
• expires-at has moved to the new certificate's expiry.
• The keystore hash inside the container is unchanged. The hash on the node path is different.
• The container only sees the new certificate after the container itself is restarted.

Expected

The annotation always matches the certificate the container can actually read, so the restart controller evicts the Pod before that certificate expires.

Possible solution

No response

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None