Kyverno Policy Engine

Purpose: For platform engineers, shows how to enable/disable Kyverno policies, create custom rules, and manage policy enforcement.

What Kyverno Does

Kyverno is a Kubernetes-native policy engine that validates, mutates, and generates resources based on ClusterPolicy rules. openCenter deploys a default ruleset of 17 ClusterPolicies that enforce pod security baselines, resource limits, and image policies. Kyverno works alongside Pod Security Admission (configured by Kubespray) to provide defense-in-depth policy enforcement.

How It's Deployed

Kyverno is deployed via FluxCD from openCenter-gitops-base as two components:

1. Kyverno engine — The admission controller and background scanner.
2. default-ruleset — 17 ClusterPolicies in applications/base/services/kyverno/default-ruleset/.

openCenter-gitops-base/applications/base/services/kyverno/
├── namespace.yaml
├── source.yaml
├── helmrelease.yaml
├── helm-values/
│   └── hardened-values.yaml
└── default-ruleset/
    ├── disallow-privileged-containers.yaml
    ├── disallow-host-namespaces.yaml
    ├── disallow-host-path.yaml
    ├── require-run-as-nonroot.yaml
    ├── restrict-seccomp.yaml
    ├── restrict-volume-types.yaml
    └── ... (17 policies total)

Customer overlay:

applications/overlays/<cluster>/services/kyverno/
├── kustomization.yaml
└── override-values.yaml

Default Ruleset

The 17 ClusterPolicies in the default ruleset enforce baseline pod security standards:

Policy	Action	What It Enforces
disallow-privileged-containers	Validate	Blocks privileged: true
disallow-host-namespaces	Validate	Blocks hostPID, hostIPC, hostNetwork
disallow-host-path	Validate	Blocks hostPath volume mounts
require-run-as-nonroot	Validate	Requires runAsNonRoot: true
restrict-seccomp	Validate	Requires RuntimeDefault or Localhost seccomp profile
restrict-volume-types	Validate	Limits allowed volume types
disallow-capabilities	Validate	Blocks added Linux capabilities
disallow-privilege-escalation	Validate	Requires allowPrivilegeEscalation: false
require-resource-limits	Validate	Requires CPU and memory limits
restrict-image-registries	Validate	Limits pull sources to approved registries

The remaining policies cover additional baseline and restricted controls.

Key Configuration

Enforcement Modes

Each ClusterPolicy has a validationFailureAction that controls behavior:

• Enforce — Blocks non-compliant resources (admission denied).
• Audit — Allows the resource but logs a violation in PolicyReport.

To switch a policy to audit mode for testing:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-run-as-nonroot
spec:
  validationFailureAction: Audit  # Changed from Enforce

Adding Policy Exceptions

When a workload legitimately needs an exception (e.g., a CNI plugin that requires hostNetwork), create a PolicyException:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: calico-host-network
  namespace: kyverno
spec:
  exceptions:
    - policyName: disallow-host-namespaces
      ruleNames:
        - host-namespaces
  match:
    any:
      - resources:
          namespaces:
            - calico-system
          names:
            - calico-node-*

Writing Custom Policies

Add custom ClusterPolicies in your cluster overlay:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-team-label
      match:
        any:
          - resources:
              kinds:
                - Deployment
                - StatefulSet
      validate:
        message: "The label 'team' is required."
        pattern:
          metadata:
            labels:
              team: "?*"

Verification

# Check Kyverno pods
kubectl get pods -n kyverno

# List all ClusterPolicies and their status
kubectl get clusterpolicies

# View policy violations (audit mode)
kubectl get policyreports -A
kubectl get clusterpolicyreports

# Test a policy without deploying (dry-run)
kubectl apply --dry-run=server -f test-pod.yaml

# Check HelmRelease status
flux get helmreleases -n flux-system | grep kyverno

Troubleshooting

Pods blocked unexpectedly: Check which policy is blocking the resource:

kubectl get events --field-selector reason=PolicyViolation -A

Create a PolicyException for legitimate workloads that need elevated privileges.

Kyverno webhook timeout: If the Kyverno admission webhook is slow or unavailable, pods across the cluster may fail to schedule. Check Kyverno pod health and resource usage. The base values include failurePolicy: Ignore on the webhook to prevent cluster-wide outages.

Common Customizations

• Registry allowlist: Edit restrict-image-registries to add your Harbor instance or other approved registries.
• Resource limit defaults: Add a mutate policy that injects default resource limits for pods that don't specify them.
• Namespace exclusions: Use exclude.namespaces in policy rules to skip system namespaces like kube-system.
• Background scanning: Kyverno scans existing resources periodically. Adjust background: true/false per policy to control this.