Skip to main content

Telemetry (OpenTelemetry)

Purpose: For platform engineers, shows how to configure OpenTelemetry Collector pipeline and auto-instrumentation.

Task Summary​

The OpenTelemetry Collector is the central telemetry pipeline in openCenter. It receives traces, metrics, and logs via OTLP from instrumented applications and routes them to the appropriate backends (Tempo, Prometheus, Loki). This guide covers Collector configuration, OTLP endpoints, and auto-instrumentation setup.

Prerequisites​

  • OpenTelemetry Collector deployed via FluxCD from openCenter-gitops-base
  • Target backends running (Prometheus, Loki, Tempo)
  • kubectl access to the cluster

OTLP Endpoints​

Applications send telemetry to the Collector's OTLP endpoints:

ProtocolEndpointPort
gRPCopentelemetry-collector.monitoring.svc.cluster.local4317
HTTPopentelemetry-collector.monitoring.svc.cluster.local4318

Configure your application's OTLP exporter to point to one of these endpoints.

Collector Pipeline Architecture​

The Collector processes telemetry through a pipeline of receivers, processors, and exporters:

Receivers          Processors              Exporters
─────────          ──────────              ─────────
otlp (gRPC/HTTP) → batch                → otlp/tempo (traces)
                  → memory_limiter       → prometheusremotewrite (metrics)
                  → k8sattributes        → loki (logs)
                  → resource

Key Processors​

batch: Groups telemetry into batches before exporting, reducing network overhead. Default: 200ms timeout, 8192 max batch size.

memory_limiter: Prevents the Collector from consuming excessive memory. Drops data if the limit is reached.

k8sattributes: Enriches telemetry with Kubernetes metadata (pod name, namespace, node, labels) by querying the Kubernetes API.

Configure the Collector​

Override Collector settings in the customer overlay:

# applications/overlays/<cluster>/services/opentelemetry-collector/override-values.yaml
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  processors:
    batch:
      timeout: 200ms
      send_batch_size: 8192
    memory_limiter:
      check_interval: 5s
      limit_mib: 512
      spike_limit_mib: 128
    k8sattributes:
      extract:
        metadata:
          - k8s.namespace.name
          - k8s.pod.name
          - k8s.node.name
  exporters:
    otlp/tempo:
      endpoint: tempo.monitoring.svc.cluster.local:4317
      tls:
        insecure: true
    prometheusremotewrite:
      endpoint: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/api/v1/write
    loki:
      endpoint: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlp/tempo]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [prometheusremotewrite]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [loki]

Auto-Instrumentation​

OpenTelemetry auto-instrumentation injects tracing libraries into application pods without code changes. This requires the OpenTelemetry Operator.

Step 1: Create an Instrumentation resource​

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: auto-instrumentation
  namespace: my-app
spec:
  exporter:
    endpoint: http://opentelemetry-collector.monitoring.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"

Step 2: Annotate your deployment​

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"    # For Java apps
    # instrumentation.opentelemetry.io/inject-python: "true" # For Python apps
    # instrumentation.opentelemetry.io/inject-nodejs: "true"  # For Node.js apps

The Operator injects an init container that adds the instrumentation agent to the pod.

Verification​

# Check Collector pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=opentelemetry-collector

# Check Collector logs for export errors
kubectl logs -n monitoring -l app.kubernetes.io/name=opentelemetry-collector --tail=20

# Verify OTLP endpoint is reachable from an app namespace
kubectl run otel-test --image=busybox --rm -it --restart=Never -n my-app -- \
  wget -qO- --timeout=3 http://opentelemetry-collector.monitoring.svc.cluster.local:4318/v1/traces

Further Reading​