Data Flow: Observability

Purpose: For platform engineers, explains how telemetry data (metrics, logs, traces) flows from workloads to Grafana.

Flow Summary

Components

Component	Namespace	Signal	Role
OpenTelemetry Collector	observability	All	Receives OTLP data from apps, forwards to Kafka
Kafka	observability	All	Event streaming buffer between collectors and backends
Prometheus	observability	Metrics	Stores time-series data, evaluates alerting rules
Alertmanager	observability	Alerts	Evaluates rules, routes notifications
kube-state-metrics	observability	Metrics	Exposes Kubernetes object state as Prometheus metrics
Promtail	observability	Logs	DaemonSet that tails container log files
Loki	observability	Logs	Log aggregation with label-based indexing
Tempo	observability	Traces	Distributed trace storage
Grafana	observability	All	Unified query and visualization across all signals

Sequence

Metrics

1. Applications expose a /metrics endpoint (Prometheus format) or emit OTLP metrics.
2. OpenTelemetry Collector scrapes or receives metrics.
3. Collector exports to Kafka topics.
4. Prometheus consumes from Kafka and stores time-series.
5. Alertmanager evaluates PrometheusRule resources and fires alerts.
6. Grafana queries Prometheus via PromQL.

Logs

1. Containers write to stdout/stderr.
2. Promtail (DaemonSet) tails log files, attaches Kubernetes labels.
3. Promtail pushes log streams to Kafka.
4. Loki consumes from Kafka and indexes logs.
5. Grafana queries Loki via LogQL.

Traces

1. Applications instrumented with OpenTelemetry SDK export spans via OTLP.
2. OpenTelemetry Collector receives, batches, and exports to Kafka.
3. Tempo consumes from Kafka and stores traces.
4. Grafana queries Tempo via TraceQL.

Related

• Logical Diagram — full cluster architecture