Skip to main content

Dashboards & Alerts

Purpose: For platform engineers, shows how to use pre-built Grafana dashboards and configure alert routing.

Task Summary​

openCenter deploys Grafana with pre-configured dashboards for cluster health, node metrics, pod resources, and platform services. Alertmanager handles alert routing, grouping, and notification delivery. This guide covers accessing dashboards, adding custom ones, and configuring alert routes.

Prerequisites​

  • kube-prometheus-stack deployed (includes Grafana and Alertmanager)
  • kubectl access to the cluster

Access Grafana​

# Port-forward to Grafana
kubectl port-forward svc/kube-prometheus-stack-grafana -n monitoring 3000:80
# Open http://localhost:3000
# Default credentials are configured in the HelmRelease values

If Keycloak SSO is configured, Grafana uses OIDC authentication and no local credentials are needed.

Pre-Built Dashboards​

kube-prometheus-stack includes dashboards deployed as ConfigMaps:

DashboardScopeKey Metrics
Kubernetes / ClusterCluster-wideCPU, memory, pod count, node status
Kubernetes / NodesPer nodeCPU, memory, disk, network per node
Kubernetes / PodsPer podCPU/memory requests vs limits, restarts
DashboardScopeKey Metrics
Kubernetes / NamespacesPer namespaceResource usage by namespace
CoreDNSDNSQuery rate, latency, errors
etcdControl planeLeader elections, WAL fsync, DB size
PrometheusSelf-monitoringScrape duration, TSDB stats, rule evaluation

These dashboards are read-only (provisioned from ConfigMaps). Changes made in the Grafana UI are lost on pod restart.

Add a Custom Dashboard​

To persist a custom dashboard, deploy it as a ConfigMap with the grafana_dashboard label:

Step 1: Export the dashboard JSON​

Design the dashboard in Grafana's UI, then export it: Dashboard settings → JSON Model → Copy.

Step 2: Create a ConfigMap​

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"  # Grafana sidecar picks up ConfigMaps with this label
data:
  my-app-dashboard.json: |
    {
      "dashboard": {
        "title": "My App Overview",
        "panels": [ ... ]
      }
    }

Step 3: Deploy via GitOps​

Add the ConfigMap to the customer overlay and commit. FluxCD deploys it, and Grafana's sidecar container loads the dashboard within 60 seconds.

Alertmanager Configuration​

Alertmanager receives alerts from Prometheus and routes them to notification channels.

Alert Routing​

Configure routes in the HelmRelease values:

# applications/overlays/<cluster>/services/kube-prometheus-stack/override-values.yaml
alertmanager:
  config:
    route:
      receiver: default
      group_by: [alertname, namespace]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - match:
            severity: critical
          receiver: pagerduty
        - match:
            severity: warning
          receiver: slack
    receivers:
      - name: default
        webhook_configs:
          - url: http://alertmanager-webhook.monitoring.svc.cluster.local:9095/webhook
      - name: slack
        slack_configs:
          - api_url: https://hooks.slack.com/services/T00/B00/XXXX
            channel: "#alerts"
            title: '{{ .GroupLabels.alertname }}'
            text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
      - name: pagerduty
        pagerduty_configs:
          - service_key: <pagerduty-integration-key>

Silencing Alerts​

Silence alerts during maintenance via the Alertmanager UI:

kubectl port-forward svc/kube-prometheus-stack-alertmanager -n monitoring 9093:9093
# Open http://localhost:9093/#/silences — create a new silence

Verification​

# Check Alertmanager is running
kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager

# View active alerts
kubectl port-forward svc/kube-prometheus-stack-alertmanager -n monitoring 9093:9093
# Open http://localhost:9093/#/alerts

# Check Grafana sidecar loaded dashboards
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana -c grafana-sc-dashboard --tail=10

Further Reading​