Skip to main content

Service Version Upgrades

Purpose: For operators, shows how to bump gitops-base tag, test in staging, and roll out service upgrades.

How Service Upgrades Work​

Platform services (cert-manager, Kyverno, Velero, etc.) are deployed via HelmRelease manifests in openCenter-gitops-base. Each service pins a specific Helm chart version in its HelmRelease. Upgrading a service means bumping that version in the base repository, tagging a new release, and updating the GitRepository ref in your cluster overlay.

The flow: update base → tag release → update cluster overlay → FluxCD reconciles → Helm controller upgrades the release.

Prerequisites​

  • Write access to the openCenter-gitops-base repository
  • A staging cluster to validate changes before production
  • flux CLI installed locally

Step 1: Update the HelmRelease in gitops-base​

In the openCenter-gitops-base repository, update the chart version for the target service:

# applications/base/services/cert-manager/helmrelease.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  chart:
    spec:
      chart: cert-manager
      version: "1.16.0"  # Updated from 1.15.3
      sourceRef:
        kind: HelmRepository
        name: jetstack

If the service has hardened values, update those too:

# applications/base/services/cert-manager/helm-values/hardened-values-v1.16.0.yaml
# Updated values for the new chart version

Commit, push, and open a PR against gitops-base. After review and merge, tag the new release:

git tag v1.1.0
git push origin v1.1.0

Step 2: Update the Cluster Overlay​

In the customer GitOps repository, update the GitRepository ref to point to the new tag:

# applications/overlays/<cluster>/services/sources/opencenter-cert-manager.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: opencenter-cert-manager
  namespace: flux-system
spec:
  url: ssh://git@github.com/opencenter-cloud/openCenter-gitops-base.git
  ref:
    tag: v1.1.0  # Updated from v1.0.0
  secretRef:
    name: opencenter-base

Commit this change to a branch, open a PR, and merge after approval.

Step 3: Validate in Staging First​

Apply the tag update to a staging cluster before production. Monitor the rollout:

# Watch the HelmRelease status
flux get helmreleases -n cert-manager -w

# Check pod rollout
kubectl rollout status deployment/cert-manager -n cert-manager

# Verify the new version is running
kubectl get pods -n cert-manager -o jsonpath='{.items[*].spec.containers[*].image}'

Step 4: Roll Out to Production​

After staging validation, merge the same GitRepository tag update to the production cluster overlay. FluxCD picks up the change and performs the upgrade automatically.

Verification​

# Confirm HelmRelease shows the new chart version
flux get helmreleases -A

# Check for any reconciliation errors
flux get kustomizations -A

# Verify service health (example: cert-manager)
kubectl get pods -n cert-manager
kubectl get certificates -A

Rollback​

If an upgrade causes issues, revert the GitRepository tag to the previous version:

git revert HEAD  # Reverts the tag bump commit
git push origin main

FluxCD reconciles and Helm controller rolls back to the previous chart version. For immediate rollback without waiting for reconciliation:

flux reconcile source git opencenter-cert-manager -n flux-system
flux reconcile helmrelease cert-manager -n cert-manager

Troubleshooting​

  • HelmRelease stuck in Upgrading — Check Helm controller logs: kubectl logs -n flux-system deploy/helm-controller --tail=50. Common cause: invalid values or CRD incompatibility.
  • Pods crash-looping after upgrade — Check pod logs and events. The new chart version may require updated override values in the cluster overlay.
  • Source not found — Verify the tag exists in the gitops-base repository and the GitRepository secret has valid SSH credentials.