Skip to main content

Telelemetry

Integration with Prometheus​

OBLV Deploy installs an OTel collector in Kubernetes. This OTel collector can be configured with multiple backends for telemetry data.

For metrics, we can configure the OTel collector to send the metrics to the Prometheus server.

  1. Export the collector config map into a file:

    kubectl get cm oblv-deploy-stack-oblv-deploy-chart-collector-config \
    		-n kube-system -o yaml > /tmp/collector-config.yaml
    	
    cat /tmp/collector-config.yaml

    apiVersion: v1
    data:
      collector-config.yaml: |
        receivers:
            otlp:
                protocols:
                    grpc:
                        endpoint: 0.0.0.0:4317
                    http:
                        endpoint: 0.0.0.0:4318
        exporters:
            debug:
                verbosity: detailed
        service:
            pipelines:
                traces:
                    receivers: [otlp]
                    exporters: [debug]
                metrics:
                    receivers: [otlp]
                    exporters: [debug]
                logs:
                    receivers: [otlp]
                    exporters: [debug]
    kind: ConfigMap

  2. Edit the config map to configure Prometheus as an exporter. Prometheus prefers to scrape the metrics from a source. So, we will configure the OTel collector to expose an endpoint for Prometheus to scrape:


    apiVersion: v1
    data:
      collector-config.yaml: |
        receivers:
            otlp:
                protocols:
                    grpc:
                        endpoint: 0.0.0.0:4317
                    http:
                        endpoint: 0.0.0.0:4318
        exporters:
            debug:
                verbosity: detailed
            prometheus:
                endpoint: 0.0.0.0:8889
                send_timestamps: true
        service:
            pipelines:
                traces:
                    receivers: [otlp]
                    exporters: [debug]
                metrics:
                    receivers: [otlp]
                    exporters: [debug, prometheus]
                logs:
                    receivers: [otlp]
                    exporters: [debug]
    kind: ConfigMap

  3. Update the configmap:

    kubectl apply -f /tmp/collector-config.yaml -n kube-system

    configmap/oblv-deploy-stack-oblv-deploy-chart-collector-config configured

  4. Restart the Otel collector pod:

      kubectl rollout restart \
    	  deployment/oblv-deploy-stack-oblv-deploy-chart-collector \
    	  -n kube-system
    	
    	deployment.apps/oblv-deploy-stack-oblv-deploy-chart-collector restarted

  5. Lastly, we can update our Prometheus config to scrape from this endpoint:

    global:
      evaluation_interval: 30s
      scrape_interval: 60s
    scrape_configs:
    - job_name: otel-collector
      static_configs:
      - targets:
        - 'a548e65e751f74a19890f5774a1cd99b-8e43192973cc255f.elb.eu-central-1.amazonaws.com:8889'
      honor_labels: true
      honor_timestamps: true