Skip to main content

Secrets & Decryption

Purpose: For operators, shows how to resolve SOPS decryption failures, missing Age key secrets, and key rotation issues.

Prerequisites​

  • kubectl access to the cluster
  • sops CLI installed
  • Access to the Age key file in secrets/age/

SOPS Decryption Fails in FluxCD​

Symptom: FluxCD Kustomization shows failed to decrypt in its status.

Step 1: Verify the Age Key Secret​

kubectl get secret sops-age -n flux-system

If the secret is missing, recreate it from the local key file:

kubectl create secret generic sops-age \
  --from-file=age.agekey=secrets/age/<cluster>_keys.txt \
  -n flux-system --dry-run=client -o yaml | kubectl apply -f -

Step 2: Verify the Key Matches​

The Age public key in .sops.yaml must match the private key in the secret. Extract the public key:

grep "public key" secrets/age/<cluster>_keys.txt

Compare with the recipient in .sops.yaml:

grep "age1" applications/overlays/<cluster>/.sops.yaml

If they differ, the secrets were encrypted with a different key. Re-encrypt with the current key:

sops updatekeys applications/overlays/<cluster>/services/keycloak/secret.yaml

Step 3: Force Reconciliation​

flux reconcile kustomization <name> -n flux-system

Cannot Decrypt Locally with sops​

Symptom: sops -d <file> fails with could not decrypt data key.

The Age key is not in the default location. Export it:

export SOPS_AGE_KEY_FILE=secrets/age/<cluster>_keys.txt
sops -d applications/overlays/<cluster>/services/keycloak/secret.yaml

Key Rotation Issues​

Dual-Key Period​

During rotation, both old and new keys are valid. The CLI uses a dual-key strategy:

  1. opencenter cluster rotate-keys generates a new key pair
  2. Re-encrypts all secrets with both old and new keys
  3. Updates the sops-age Kubernetes secret with the new key
  4. Old key remains valid until the next rotation

Rotation Fails Mid-Way​

If rotation is interrupted, some files may be encrypted with the new key while others still use the old key.

Check which key each file uses:

sops filestatus applications/overlays/<cluster>/services/keycloak/secret.yaml

Re-run rotation to complete:

opencenter cluster rotate-keys <cluster>

Key Expiration Warnings​

Check key age:

opencenter cluster check-keys <cluster>

Age keys older than 90 days and SSH keys older than 180 days trigger warnings. Rotate before they expire to avoid disruption.

Secrets Sync Failures​

Symptom: opencenter cluster sync-secrets reports drift.

Validate secrets consistency first:

opencenter cluster validate-secrets <cluster>

This compares the local encrypted files against what the cluster expects. If drift is detected, sync resolves it:

opencenter cluster sync-secrets <cluster>