Telemetry

Multiple Backends

When we install the helm chart for OBLV Deploy, it installs an OTel collector and a Prometheus backend.

Users can configure this Otel collector to send telemetry data in multiple backends.

1

We can edit this config file to add multiple backends for different telemetry data. We can add Loki backend for logs and a Jaeger backend for traces:

apiVersion: v1
data:
collector-config.yaml: |
    receivers:
        otlp:
            protocols:
                grpc:
                    endpoint: 0.0.0.0:4317
                http:
                    endpoint: 0.0.0.0:4318
    exporters:
        debug:
            verbosity: detailed
        prometheus: # pull based exporter
            endpoint: 0.0.0.0:8889
            send_timestamps: true
        otlp/jaeger:
                    endpoint: https://jaeger.oblv.com:4317
                loki:
                    endpoint: https://loki.example.com:3100/loki/api/v1/push
    service:
        pipelines:
            traces:
                receivers: [otlp]
                exporters: [debug, otlp/jaeger]
            metrics:
                receivers: [otlp]
                exporters: [debug, prometheus]
            logs:
                receivers: [otlp]
                exporters: [debug, loki]
kind: ConfigMap

2

Deploy the Collector:

Apply a Deployment that mounts this config and runs the collector:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:0.94.0
        resources:
          limits:
            memory: "256Mi"
            cpu: "0.5"
          requests:
            memory: "128Mi"
            cpu: "0.25"
        ports:
        - containerPort: 4317
          name: otlp-grpc
        - containerPort: 4318
          name: otlp-http
        - containerPort: 8889
          name: prom-exp-http
        command:
        - "/otelcol"
        - "--config=/etc/otel/collector-config.yaml"
        volumeMounts:
        - name: otel-collector-config-vol
          mountPath: /etc/otel
      volumes:
      - name: otel-collector-config-vol
        configMap:
          name: otel-collector-config
          items:
          - key: collector-config.yaml
            path: collector-config.yaml