Edit

Purpose: For platform engineers and developers, shows how to systematically verify services deployed on an openCenter Kind cluster, following the FluxCD dependency chain.

Prerequisites

  • A running Kind cluster created via opencenter cluster deploy

  • kubectl, flux CLI tools installed

  • Access to the cluster’s kubeconfig

Set up your environment:

CLUSTER_NAME="my-cluster"
GITOPS_DIR=$(opencenter cluster describe "$CLUSTER_NAME" 2>/dev/null | grep "git_dir:" | awk '{print $2}')
export KUBECONFIG="$GITOPS_DIR/infrastructure/clusters/$CLUSTER_NAME/kubeconfig.yaml"

Service Dependency Graph

Services must be verified in dependency order. The graph below shows the FluxCD Kustomization dependencies:

                    ┌─────────────────┐
                    │   flux-system   │
                    │  (GitRepository)│
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │     sources     │
                    │ (GitRepositories│
                    │  for all svcs)  │
                    └────────┬────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
┌───────▼───────┐   ┌────────▼────────┐   ┌───────▼───────┐
│  cert-manager │   │   gateway-api   │   │    metallb    │
│    (base)     │   │ (envoy-gateway) │   │    (base)     │
└───────┬───────┘   └────────┬────────┘   └───────┬───────┘
        │                    │                    │
        └───────────┬────────┴────────────────────┘
                    │
           ┌────────▼────────┐
           │     gateway     │
           │ (depends on     │
           │  cert-manager & │
           │  gateway-api)   │
           └────────┬────────┘
                    │
        ┌───────────┴─────────────────────────────┐
        │                                         │
┌───────▼───────┐                         ┌───────▼───────┐
│   headlamp    │                         │observability- │
│               │                         │  namespace    │
└───────────────┘                         └───────┬───────┘
                                                  │
                                          ┌───────▼───────┐
                                          │kube-prometheus│
                                          │    -stack     │
                                          └───────┬───────┘
                                                  │
                                          ┌───────▼───────┐
                                          │     loki      │
                                          └───────────────┘

Verification Phases

Phase 0: Pre-Flight Checks

Verify cluster infrastructure before checking services.

# Check nodes
kubectl get nodes
# Expected: All nodes Ready

# Check system pods
kubectl get pods -n kube-system
# Expected: All pods Running

# Check Gitea connectivity (for local Kind clusters)
opencenter local gitea status
# Expected: Running: true, Kind Attached: true

| Check | Command | Expected | | --- | --- | --- | | Nodes | kubectl get nodes | All Ready | | System Pods | kubectl get pods -n kube-system | All Running | | Gitea | opencenter local gitea status | Running: true | | API Server | kubectl cluster-info | Reachable |

Phase 1: FluxCD Core

Verify FluxCD controllers are operational.

# Check Flux pods
kubectl get pods -n flux-system

# Check Git source
flux get sources git -n flux-system
# Expected: flux-system READY=True

# Check all kustomizations
flux get kustomizations -n flux-system

| Component | Command | Expected | | --- | --- | --- | | Source Controller | kubectl get deploy source-controller -n flux-system | READY 1/1 | | Kustomize Controller | kubectl get deploy kustomize-controller -n flux-system | READY 1/1 | | Helm Controller | kubectl get deploy helm-controller -n flux-system | READY 1/1 | | GitRepository | flux get sources git -n flux-system | READY=True |

Phase 2: Sources

Verify all service GitRepositories are ready.

# Check sources kustomization
flux get kustomization sources -n flux-system

# List all GitRepositories
flux get sources git -A

Key sources to verify:

  • opencenter-cert-manager

  • opencenter-gateway-api

  • opencenter-metallb

  • opencenter-headlamp

  • opencenter-kube-prometheus-stack (if observability enabled)

  • opencenter-observability (if observability enabled)

Phase 3: Infrastructure Services

These services form the foundation for application services.

3.1 cert-manager

# Kustomization status
flux get kustomization cert-manager-base -n flux-system

# HelmRelease status
flux get helmrelease cert-manager -n cert-manager

# Pod status
kubectl get pods -n cert-manager

# Verify CRDs installed
kubectl get crd certificates.cert-manager.io

# Check cluster issuers
kubectl get clusterissuers

| Check | Command | Expected | | --- | --- | --- | | Kustomization | flux get kustomization cert-manager-base | READY=True | | HelmRelease | flux get helmrelease cert-manager -n cert-manager | READY=True | | Pods | kubectl get pods -n cert-manager | All Running | | Webhook | kubectl get validatingwebhookconfigurations cert-manager-webhook | Exists |

3.2 gateway-api (Envoy Gateway)

# Kustomization status
flux get kustomization envoy-gateway-api-base -n flux-system

# HelmRelease status
flux get helmrelease envoy-gateway-api -n envoy-gateway-system

# Pod status
kubectl get pods -n envoy-gateway-system

# Verify GatewayClass
kubectl get gatewayclass

| Check | Command | Expected | | --- | --- | --- | | Kustomization | flux get kustomization envoy-gateway-api-base | READY=True | | HelmRelease | flux get helmrelease envoy-gateway-api -n envoy-gateway-system | READY=True | | Pods | kubectl get pods -n envoy-gateway-system | All Running | | GatewayClass | kubectl get gatewayclass | envoy-gateway exists |

3.3 metallb

# Kustomization status
flux get kustomization metallb-base -n flux-system

# HelmRelease status
flux get helmrelease metallb -n metallb-system

# Pod status
kubectl get pods -n metallb-system

# Check IP address pool
kubectl get ipaddresspool -n metallb-system

| Check | Command | Expected | | --- | --- | --- | | Kustomization | flux get kustomization metallb-base | READY=True | | HelmRelease | flux get helmrelease metallb -n metallb-system | READY=True | | Pods | kubectl get pods -n metallb-system | All Running | | IPAddressPool | kubectl get ipaddresspool -n metallb-system | Pool configured |

Phase 4: Gateway

The gateway depends on cert-manager and gateway-api.

# Kustomization status
flux get kustomization gateway -n flux-system

# Gateway resource
kubectl get gateway -A

# Check gateway has address
kubectl get gateway -n gateway-system -o wide

# HTTPRoutes
kubectl get httproutes -A

# Envoy proxy pods
kubectl get pods -n gateway-system

| Check | Command | Expected | | --- | --- | --- | | Kustomization | flux get kustomization gateway | READY=True | | Gateway | kubectl get gateway -A | Programmed=True | | Address | kubectl get gateway -n gateway-system -o jsonpath='{.items[0].status.addresses}' | Has IP | | Envoy Pods | kubectl get pods -n gateway-system | All Running |

Phase 5: Application Services

5.1 headlamp

# Kustomization status
flux get kustomization headlamp-base -n flux-system

# HelmRelease status
flux get helmrelease headlamp -n headlamp

# Pod status
kubectl get pods -n headlamp

# HTTPRoute
kubectl get httproute -n headlamp

Access test:

kubectl port-forward -n headlamp svc/headlamp 8080:80
# Open http://localhost:8080

Phase 6: Observability Stack

Skip this phase if observability services are disabled.

6.1 observability-namespace

flux get kustomization observability-namespace -n flux-system
kubectl get namespace observability

6.2 kube-prometheus-stack

# Kustomization status
flux get kustomization kube-prometheus-stack-base -n flux-system

# HelmRelease status
flux get helmrelease kube-prometheus-stack -n observability

# Component pods
kubectl get pods -n observability -l app.kubernetes.io/name=prometheus
kubectl get pods -n observability -l app.kubernetes.io/name=grafana
kubectl get pods -n observability -l app.kubernetes.io/name=alertmanager

6.3 loki

# Kustomization status
flux get kustomization loki-base -n flux-system

# HelmRelease status
flux get helmrelease loki -n observability

# Pods
kubectl get pods -n observability -l app.kubernetes.io/name=loki

Quick Health Check

Run this for a rapid cluster health assessment:

# All kustomizations should be READY=True
flux get kustomizations -A

# All HelmReleases should be READY=True
flux get helmreleases -A

# Find unhealthy pods
kubectl get pods -A | grep -v Running | grep -v Completed

Troubleshooting

GitRepository Not Ready

Symptoms: flux get sources git shows READY=False

Diagnosis:

kubectl describe gitrepository flux-system -n flux-system
opencenter local gitea status

Common causes:

  • Gitea not attached to Kind network

  • TLS certificate missing host IP as SAN

  • Network connectivity issues

Fix:

opencenter cluster deploy my-cluster --container-runtime podman --from-step gitea-attach-kind

HelmRelease Stuck Progressing

Symptoms: flux get helmrelease shows READY=Unknown or Progressing

Diagnosis:

kubectl describe helmrelease <name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Common causes:

  • Helm chart download failed

  • Resource constraints on nodes

  • Missing CRDs

Fix:

# Force reconciliation
flux reconcile helmrelease <name> -n <namespace>

# Suspend and resume
flux suspend helmrelease <name> -n <namespace>
flux resume helmrelease <name> -n <namespace>

Pods CrashLoopBackOff

Symptoms: kubectl get pods shows CrashLoopBackOff

Diagnosis:

kubectl logs <pod-name> -n <namespace> --previous
kubectl describe pod <pod-name> -n <namespace>

Common causes:

  • Configuration errors in Helm values

  • Missing secrets or configmaps

  • Resource limits too restrictive

Gateway No Address

Symptoms: Gateway shows no external IP

Diagnosis:

kubectl describe gateway -n gateway-system
kubectl get ipaddresspool -n metallb-system
kubectl logs -n metallb-system -l app.kubernetes.io/name=metallb

Common causes:

  • MetalLB not configured

  • IPAddressPool exhausted

  • L2Advertisement missing

Fix:

# Check MetalLB configuration
kubectl get ipaddresspool,l2advertisement -n metallb-system -o yaml

Kustomization Dependency Failed

Symptoms: Kustomization shows dependency 'xxx' is not ready

Diagnosis:

flux get kustomization <dependency-name> -n flux-system
kubectl describe kustomization <dependency-name> -n flux-system

Fix: Resolve the dependency first, then the dependent kustomization will reconcile automatically.

Verification Script

Save this script to run a complete verification:

#!/bin/bash
set -e

CLUSTER_NAME="${1:-my-cluster}"

echo "=========================================="
echo "openCenter Kind Cluster Verification"
echo "Cluster: $CLUSTER_NAME"
echo "=========================================="

GITOPS_DIR=$(opencenter cluster describe "$CLUSTER_NAME" 2>/dev/null | grep "git_dir:" | awk '{print $2}')
export KUBECONFIG="$GITOPS_DIR/infrastructure/clusters/$CLUSTER_NAME/kubeconfig.yaml"

echo ""
echo "=== Phase 0: Pre-Flight ==="
kubectl get nodes
opencenter local gitea status 2>/dev/null || true

echo ""
echo "=== Phase 1: FluxCD Core ==="
kubectl get pods -n flux-system
flux get sources git -n flux-system

echo ""
echo "=== Phase 2: Sources ==="
flux get sources git -A

echo ""
echo "=== Phase 3: Infrastructure ==="
for svc in cert-manager-base envoy-gateway-api-base metallb-base; do
  echo "--- $svc ---"
  flux get kustomization "$svc" -n flux-system 2>/dev/null || echo "Not deployed"
done

echo ""
echo "=== Phase 4: Gateway ==="
flux get kustomization gateway -n flux-system 2>/dev/null || echo "Not deployed"
kubectl get gateway -A 2>/dev/null || echo "No gateways"

echo ""
echo "=== Phase 5: Applications ==="
flux get kustomization headlamp-base -n flux-system 2>/dev/null || echo "Not deployed"

echo ""
echo "=== Summary ==="
echo "--- Kustomizations ---"
flux get kustomizations -A
echo ""
echo "--- HelmReleases ---"
flux get helmreleases -A

echo ""
echo "=========================================="
echo "Verification Complete"
echo "=========================================="

Service Dependency Reference

| Service | Depends On | Namespace | | --- | --- | --- | | sources | flux-system | flux-system | | cert-manager-base | sources | cert-manager | | envoy-gateway-api-base | sources | envoy-gateway-system | | metallb-base | sources | metallb-system | | gateway | cert-manager-base, envoy-gateway-api-base | gateway-system | | headlamp-base | sources | headlamp | | observability-namespace | sources | observability | | kube-prometheus-stack-base | sources, observability-namespace | observability | | loki-base | kube-prometheus-stack-base | observability |

Evidence

  • FluxCD Kustomization templates: internal/gitops/templates/cluster-apps-base/services/fluxcd/

  • Kind bootstrap provider: internal/cluster/kind_bootstrap_provider.go

  • Service dependency definitions: internal/gitops/templates/cluster-apps-base/services/fluxcd/*.yaml.tpl

  • Kind defaults: internal/config/defaults/kind.yaml