Understanding Kubernetes Pod Secrets Synchronization Issues

The complexity of Kubernetes environments, particularly regarding secret management, can lead to unexpected discrepancies among pods. A recent incident illustrated how a deployed application struggled with synchronizing its secrets, resulting in one of two pods not reflecting the latest configuration updates.

This specific deployment used Azure Key Vault alongside the Secrets Store CSI Driver. Each of the two pods was supposed to mount a specific CSI volume connected to a SecretProviderClass. This setup was meant to ensure that secret values fetched from the vault would synchronize effectively with Kubernetes secrets. Despite this, a mismatch was discovered during a standard post-deployment review – one pod had updated environment variables while the other did not.

The Discrepancy Explained

Upon inspecting the deployment, all the indicators seemed to align correctly. The kubectl get secret command displayed the latest secret, and both pod specifications correctly pointed to this secret. However, a review of the running pods revealed something critical: when executing a command to check the environment variables, one pod indicated that a specific feature credential was missing while the other confirmed its presence.

for pod in $(kubectl get pods -l app=web -o name)
do
  printf '%s: ' "$pod"
  kubectl exec "$pod" -c app -- printenv \
    | cut -d= -f1 \
    | grep -qx FEATURE_CREDENTIAL \
    && echo present || echo missing
done

Asynchronous Updates in Action

The root cause of the problem can be attributed to how the Secrets Store CSI Driver and Kubernetes handle updates. Synchronization occurs at different times. The CSI Driver updates the native Kubernetes Secret only after a pod mounts the secret provider class, whereas Kubernetes initializes its environment variables at container creation. This means if the secret is updated after the container starts, the old value remains in use without restarting the container.

At the time of the investigation, the Kubernetes secret reflected the recent update, but each container retained the values from when they were initially started. This situation exemplified why the managed secret alone cannot provide a complete picture of what configuration is actively in use within each pod.

Helm's Role in Resource Ordering

Compounding the issue, the deployment’s configuration through Helm 4.2.3 affected manifest ordering. Helm sorts resources by their kind, leading to the peculiar situation where a Deployment could be established before the associated SecretProviderClass. This discrepancy led to timing issues; the original deployment of the pod happened before the synchronized update of the secrets provider class.

As a result, pod replacements during the deployment did not reflect the latest secret keys until the next pod scheduled for update mounted the newly updated secret provider class. Although I didn't log the specific timestamps, it was evident from the sequence of events that the outgoing pods maintained outdated configurations for a brief window.

Alternative Deployment Strategy

In a separate scenario using kubectl apply, applying the provider class after a rollout led to similar discrepancies. While the deployment appeared running smoothly, the three pods retained their original environment states and lacked updated configuration until a restart of the workload. This demonstrated the ordering problem could manifest in various deployment strategies.

Introducing Verification Measures

To mitigate these issues, I implemented a post-rollout verification gate that inspects each running pod to ensure that the expected key names match. During this check, if a pod fails to confirm the key, the process attempts to restart the deployment once. Should the key be absent a second time, the deployment process fails altogether.

By preventing incomplete configurations from persisting, this method bolsters reliability. However, challenges remain. If a pod cannot be inspected via kubectl exec, the process logs a warning but continues with the verification, which might let undetected issues slip through the cracks.

Refining Health Check Strategies

The initial health checks within the deployment did not cover the integration requiring updated secrets. Even though the health check returned a 200 status, indicating everything was operational, it masked the underlying issue where one pod was missing a crucial environment variable. Therefore, it’s vital for health checks to reflect the actual operational state of every pod’s environment, not just the availability of the service endpoint.

This case serves as a stark reminder of the intricacies involved in Kubernetes configurations and the necessity for comprehensive deployment strategies that account for how resources are managed over time. Regular checks and a commitment to refining the synchronization process can enhance the overall robustness of deployments.