Skip to main content

Logging (Loki)

Purpose: For platform engineers, shows how to configure Loki storage backends, retention policies, and log pipeline.

Task Summary​

Loki aggregates logs from all pods in the cluster. Promtail (deployed as a DaemonSet) tails container log files and ships them to Loki with Kubernetes labels attached. This guide covers log querying, retention configuration, and storage backend options.

Prerequisites​

  • Loki deployed via kube-prometheus-stack or standalone HelmRelease
  • Grafana configured with Loki as a data source (default in openCenter)
  • kubectl access to the cluster

Query Logs with LogQL​

Access Grafana's Explore view and select the Loki data source. LogQL queries filter by labels and optionally parse log content.

Filter by namespace and pod​

{namespace="my-app", pod=~"my-app-.*"}

Filter by log level​

{namespace="my-app"} |= "error"

Parse JSON logs and filter by field​

{namespace="my-app"} | json | level="error" | status_code >= 500

Count errors over time​

count_over_time({namespace="my-app"} |= "error" [5m])

Rate of log lines per second​

rate({namespace="my-app"}[1m])

Retention Configuration​

Retention controls how long logs are stored before deletion. Configure in the Loki HelmRelease values:

# applications/overlays/<cluster>/services/loki/override-values.yaml
loki:
  limits_config:
    retention_period: 720h  # 30 days
  compactor:
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150

Retention is enforced by the compactor, which runs periodically and deletes chunks older than retention_period.

Storage Backends​

Loki supports multiple storage backends for chunks and index:

BackendUse CaseConfiguration Key
FilesystemDevelopment, single-nodestorage_config.filesystem
S3 / MinIOProduction, multi-nodestorage_config.aws
OpenStack SwiftOpenStack environmentsstorage_config.swift

Example S3/MinIO configuration:

loki:
  storage_config:
    aws:
      s3: s3://access_key:secret_key@endpoint/bucket_name
      s3forcepathstyle: true
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      shared_store: s3

For Longhorn-backed storage (default in openCenter), the filesystem backend with a PVC is sufficient for most clusters.

Verification​

# Check Loki pods are running
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki

# Check Promtail is running on all nodes
kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail -o wide

# Verify logs are flowing — query via Grafana Explore or:
kubectl port-forward svc/loki -n monitoring 3100:3100
curl -s "http://localhost:3100/loki/api/v1/labels" | jq .

Troubleshooting​

No logs appearing in Grafana: Check Promtail pods are running and not in CrashLoopBackOff. Verify Promtail can reach Loki:

kubectl logs -n monitoring -l app.kubernetes.io/name=promtail --tail=20

"too many outstanding requests" error: Loki is overloaded. Increase limits_config.max_query_parallelism or add Loki replicas.

High storage usage: Enable retention (see above) and verify the compactor is running. Check with:

kubectl logs -n monitoring -l app.kubernetes.io/component=compactor --tail=20

Further Reading​