Telemetry
Integrating Telemetry Plugin with OpenTelemetry Collector
In a NitroEnclaveDeployment, telemetry is handled by a dedicated telemetry plugin that runs as a special container inside the enclave. This plugin is responsible for collecting internal metrics and logs, and then pushing them to an OpenTelemetry Collector. Once the telemetry data reaches the collector, it can be processed and exported to any backend you choose (for example, Prometheus for metrics or Loki for logs).
To install OpenTelemetry Collector, check out the official documentation.
Below is a sample manifest that demonstrates how to configure the telemetry plugin for a NitroEnclaveDeployment. In this example, the telemetry plugin pushes data to an OTEL collector running at the endpoint http://otel-collector.monitoring.svc.cluster.local:4318.
apiVersion: k8s.oblv.com/v1alpha1
kind: NitroEnclaveDeployment
metadata:
name: hello-fastapi
namespace: default
spec:
userPlugins:
- name: fastapi
image: public.ecr.aws/oblivious-ai/oblv-sample-fastapi:latest
ports:
- containerPort: 8001
hostPort: 4455
command:
- "python"
- "/app/uvicorn_runner.py"
plugins:
telemetry:
image: public.ecr.aws/oblivious-ai/oblv-telemetry-dev:dev
name: telemetry-plugin
volumes:
- containerPath: /etc/oblv
readOnly: true
source:
configMap:
name: fastapi-telemetry-configmap
items:
- key: config.yaml
path: config.yaml
name: telemetry-config-vol
env:
- name: HOST_PORT
value: '8100'
- name: EXPORTER_URL
value: "http://otel-collector.monitoring.svc.cluster.local:4318"
outboundConnections:
- fqdn:
value: otel-collector.monitoring.svc.cluster.local
port: 4318
tls: false
redirects: false
replicas: 1
serviceAccount: enclave-pod
hugepages-1Gi: 12Gi
enclaveCpuCount: 2
ingress:
enabled: true
internetFacing: true
dnsHostName: monitoring-test.oblv.com
ingressTlsCertificate: oblv-ingress-tls
ports:
- port: 4455
targetPort: 4455
caCertDetails:
enclaveCertType: ENCLAVE_GENERATED
Additionally, the telemetry plugin’s configuration is provided via a ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: fastapi-telemetry-configmap
namespace: default
data:
config.yaml: 'scrape_configs: []'
Here is a sample metric for k8s_node_memory_usage_bytes
:
{
job="oblv_telemetry_service",
k8s_node_name="hello-fastapi-788791ab48-58c6dd584c-bsmds",
oblv_deployment_name="hello-fastapi"
}
2.113527808e+09 1745181459055
In this example, the label k8s_node_name
is not the name of the EKS Node hosting the enclave. It is the name of enclave itself where the static pods are running. It gets the same name as the NitroEnclave pod.