Skip to main content

FluxCD Reconciliation

Purpose: For operators, shows how to diagnose and resolve FluxCD reconciliation issues.

Prerequisites​

  • kubectl access to the cluster
  • flux CLI installed

Diagnosing Reconciliation State​

Check all FluxCD resources at once:

flux get all -A

Look for resources with Ready: False or Suspended: True.

GitRepository Authentication Failure​

Symptom: GitRepository shows ssh: handshake failed or authentication required.

flux get sources git -A

The deploy key secret may be missing or invalid.

  1. Verify the secret exists:
kubectl get secret opencenter-base -n flux-system
  1. If missing, recreate it:
flux create secret git opencenter-base \
  --ssh-key-algorithm=ed25519 \
  --url=ssh://git@github.com/opencenter-cloud/openCenter-gitops-base.git \
  -n flux-system
  1. Add the public key as a deploy key on the GitHub repository.

Kustomization Path Not Found​

Symptom: Kustomization shows path not found in source.

The spec.path in the Kustomization does not exist in the referenced GitRepository at the pinned tag.

  1. Check which tag is referenced:
kubectl get gitrepository opencenter-cert-manager -n flux-system -o jsonpath='{.spec.ref.tag}'

  1. Verify the path exists at that tag in openCenter-gitops-base.

  2. Update the tag if the path was moved in a newer release.

Kustomization Variable Substitution Failure​

Symptom: Kustomization shows var substitution failed or resources contain literal ${VARIABLE} strings.

The cluster-vars ConfigMap is missing or incomplete.

kubectl get configmap cluster-vars -n flux-system -o yaml

Add missing variables and FluxCD will re-reconcile on the next interval.

SOPS Decryption Failure​

Symptom: Kustomization shows failed to decrypt.

The sops-age secret in flux-system is missing or contains the wrong key.

kubectl get secret sops-age -n flux-system

Recreate from the local key file:

kubectl create secret generic sops-age \
  --from-file=age.agekey=secrets/age/<cluster>_keys.txt \
  -n flux-system --dry-run=client -o yaml | kubectl apply -f -

HelmRelease Stuck in Installing​

Symptom: HelmRelease shows install retries exhausted.

flux get helmreleases -A
kubectl describe helmrelease <name> -n <namespace>

Common causes:

  • Missing CRDs (install the CRD Kustomization first)
  • Invalid Helm values (check events for rendering errors)
  • Namespace does not exist

Force a retry after fixing the root cause:

flux reconcile helmrelease <name> -n <namespace>

Forcing Reconciliation​

To trigger immediate reconciliation without waiting for the interval:

# Reconcile a specific source
flux reconcile source git opencenter-cert-manager -n flux-system

# Reconcile a kustomization
flux reconcile kustomization cert-manager-base -n flux-system

# Reconcile a helm release
flux reconcile helmrelease cert-manager -n cert-manager

Suspending and Resuming​

To pause reconciliation during maintenance:

flux suspend kustomization cert-manager-base -n flux-system
# ... perform maintenance ...
flux resume kustomization cert-manager-base -n flux-system