Skip to main content

Secrets Model

Purpose: For platform engineers, explains the secrets flow from plaintext in CLI config through SOPS encryption to Kubernetes Secrets.

Concept Summary​

openCenter treats the CLI configuration file as the single source of truth for all cluster secrets. Secrets never exist in plaintext in Git. Instead, they pass through a pipeline: plaintext in the operator's local config → SOPS encryption with Age keys → committed to Git as ciphertext → decrypted by FluxCD during reconciliation → stored as Kubernetes Secrets (encrypted at rest in etcd).

This model provides dual encryption: secrets are encrypted in the Git repository (SOPS) and encrypted at rest in the Kubernetes cluster (etcd encryption). At no point does a secret travel over the wire or sit on disk in plaintext outside the operator's workstation.

How It Works​

The Configuration File​

The openCenter CLI stores cluster configuration at:

~/.config/opencenter/clusters/<org>/<cluster>/.<cluster>-config.yaml

This file contains plaintext values for service credentials, API tokens, and other sensitive data. It never leaves the operator's machine and is excluded from Git.

SOPS Encryption​

When the operator runs opencenter cluster sync-secrets, the CLI:

  1. Reads plaintext values from the config file
  2. Generates Kubernetes Secret manifests
  3. Encrypts them using SOPS with the cluster's Age public key
  4. Writes encrypted files to the GitOps repository (safe to commit)

The encrypted files use SOPS's encrypted_regex feature to encrypt only the data and stringData fields, leaving metadata (name, namespace, labels) in plaintext for GitOps tooling to read.

Git as Encrypted Storage​

Encrypted secret files live alongside other manifests in the customer repository:

applications/overlays/<cluster>/services/keycloak/secret.yaml  # SOPS-encrypted
applications/overlays/<cluster>/services/harbor/secret.yaml    # SOPS-encrypted

Because only values are encrypted, git diff still shows which secrets changed (by filename and metadata), even though the actual values are opaque.

FluxCD Decryption​

FluxCD Kustomizations are configured with SOPS decryption:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age  # Contains the Age private key

During reconciliation, FluxCD decrypts SOPS-encrypted files on-the-fly using the Age private key stored in the sops-age Secret in the flux-system namespace. The decrypted Kubernetes Secrets are then applied to the cluster.

etcd Encryption at Rest​

Kubernetes is configured (via Kubespray's k8s_hardening.yml) to encrypt Secret resources at rest in etcd. This means secrets are encrypted both in Git (SOPS) and in the cluster's backing store (etcd encryption).

Trade-offs and Alternatives​

Why SOPS + Age instead of Vault or Sealed Secrets?

  • SOPS files are standard YAML — no custom CRDs or controllers beyond FluxCD
  • Age keys are simple files, no server infrastructure to maintain
  • Works identically in air-gapped environments (no external service dependency)
  • FluxCD has native SOPS support, so decryption is part of the reconciliation loop

The trade-off is that key distribution is manual (the Age private key must be placed in the cluster as a Kubernetes Secret). For clusters that need dynamic secret generation or lease-based credentials, HashiCorp Vault is a complementary option but is not part of the default openCenter stack.

Why dual encryption?

Git encryption (SOPS) protects secrets if the repository is compromised. etcd encryption protects secrets if the cluster's storage is compromised. Neither alone covers both threat vectors.

Common Misconceptions​

"The config file is the secret store." The config file is the source of truth for secret values, but it is not a secret store. It has no access control, audit logging, or versioning beyond what the filesystem provides. Operators should protect it with filesystem permissions and rotate credentials if the workstation is compromised.

"FluxCD stores decrypted secrets in memory." FluxCD decrypts secrets only during the apply phase. The decrypted values are passed directly to the Kubernetes API server. FluxCD does not cache decrypted secrets.

"Changing a secret in the config file updates the cluster." Changing the config file alone does nothing. The operator must run opencenter cluster sync-secrets to re-encrypt and commit, then FluxCD reconciles the change to the cluster.

Further Reading​