Skip to main content

Key Rotation

Purpose: For platform engineers, shows how to perform zero-downtime dual-key rotation for Age keys (90-day) and SSH keys (180-day).

Task Summary​

openCenter manages two types of encryption keys with distinct lifecycle policies:

Key TypeLocationRotation PeriodImpact if Expired
SOPS Age keysecrets/age/<cluster>_keys.txt90 daysFluxCD cannot decrypt new secrets
SSH deploy keysecrets/ssh/180 daysFluxCD cannot pull from Git repositories

Both use a dual-key rotation strategy: the new key is added alongside the old key, secrets are re-encrypted, and the old key is removed only after verification. This prevents any window where decryption fails.

Prerequisites​

  • opencenter CLI installed and configured
  • Access to the cluster's configuration directory
  • kubectl access to the target cluster (for verification)
  • sops CLI installed (for manual verification)

Check Key Expiration​

Run check-keys before any rotation to see current key ages and upcoming expirations:

opencenter cluster check-keys <cluster-name>

Expected output:

Age key: created 2025-04-15, age 78 days, expires in 12 days
SSH key: created 2025-02-01, age 151 days, expires in 29 days

Keys within 14 days of expiration are flagged with a warning.

Rotate SOPS Age Keys​

Step 1: Run the rotation command​

opencenter cluster rotate-keys <cluster-name> --type age

This command:

  1. Generates a new Age keypair
  2. Adds the new public key to .sops.yaml creation rules (dual-key period)
  3. Re-encrypts all SOPS-encrypted files with both old and new keys
  4. Updates the sops-age Kubernetes Secret in flux-system namespace

Step 2: Verify re-encryption​

Confirm that encrypted files reference both keys:

sops --decrypt applications/overlays/<cluster>/services/keycloak/secret.yaml > /dev/null
echo $?  # Should return 0

Step 3: Verify FluxCD decryption​

Force a reconciliation and check that Kustomizations succeed:

flux reconcile kustomization flux-system --with-source
flux get kustomizations

All Kustomizations should show Ready: True. If any show decryption errors, the sops-age Secret in the cluster may not contain the new key yet.

Step 4: Remove the old key​

After confirming all secrets decrypt correctly (wait at least one full reconciliation cycle):

opencenter cluster rotate-keys <cluster-name> --type age --finalize

This removes the old key from .sops.yaml and re-encrypts files with only the new key.

Rotate SSH Deploy Keys​

opencenter cluster rotate-keys <cluster-name> --type ssh

This command:

  1. Generates a new Ed25519 SSH keypair
  2. Outputs the new public key for adding to Git hosting (GitHub/Gitea deploy keys)
  3. Updates the FluxCD GitRepository secret in the cluster

After running, add the new public key to your Git provider as a deploy key, then verify:

flux reconcile source git opencenter-base
flux get sources git

Remove the old deploy key from the Git provider once all sources show Ready: True.

Verification​

After any rotation, run the full validation suite:

opencenter cluster validate-secrets <cluster-name>

This checks:

  • All SOPS-encrypted files decrypt with the current key
  • FluxCD sops-age Secret matches the local key
  • No orphaned references to old keys in .sops.yaml

Troubleshooting​

FluxCD shows "failed to decrypt" after Age rotation: The sops-age Secret in the cluster may be stale. Recreate it:

kubectl create secret generic sops-age \
  --from-file=age.agekey=secrets/age/<cluster>_keys.txt \
  -n flux-system --dry-run=client -o yaml | kubectl apply -f -

SSH source shows authentication failure after rotation: Confirm the new public key was added to the Git provider. Check the FluxCD secret:

kubectl get secret opencenter-base -n flux-system -o jsonpath='{.data.identity}' | base64 -d | head -1

The key fingerprint should match the newly generated key.