Receiving telemetry data

After setting up the OpenTelemetry Collector and instrumenting your application, you need to connect the instrumentation and OpenTelemetry Collector so that the OpenTelemetry Collector can receive telemetry data from the instrumentation.

Receiving telemetry data from multiple clusters

If you need the Collector to receive telemetry data from multiple remote clusters, create one OpenTelemetry Collector instance in each one of the remote clusters, and then have all of their telemetry data forwarded to a central OpenTelemetry Collector instance.

Prerequisites

The Red Hat build of OpenTelemetry Operator is installed.
The Tempo Operator is installed.
A TempoStack instance is deployed on the cluster.
The following mounted certificates: Issuer, self-signed certificate, CA issuer, client and server certificates. To create any of these certificates, see step 1.

Procedure

Mount the following certificates in the OpenTelemetry Collector instance, skipping already mounted certificates.

An Issuer to generate the certificates by using the cert-manager Operator for Red Hat OpenShift.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

A self-signed certificate.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca
spec:
  isCA: true
  commonName: ca
  subject:
    organizations:
      - <your_organization_name>
    organizationalUnits:
      - Widgets
  secretName: ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io

A CA issuer.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-ca-issuer
spec:
  ca:
    secretName: ca-secret

The client and server certificates.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: server
spec:
  secretName: server-tls
  isCA: false
  usages:
    - server auth
    - client auth
  dnsNames:
  - "otel.observability.svc.cluster.local" 
  issuerRef:
    name: ca-issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: client
spec:
  secretName: client-tls
  isCA: false
  usages:
    - server auth
    - client auth
  dnsNames:
  - "otel.observability.svc.cluster.local" 
  issuerRef:
    name: ca-issuer

List of exact DNS names to be mapped to a solver in the server OpenTelemetry Collector instance.
List of exact DNS names to be mapped to a solver in the client OpenTelemetry Collector instance.

Create a service account for the OpenTelemetry Collector instance.
Example ServiceAccount
```
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector-deployment
```

Create a cluster role for the service account.

Example ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  
  
- apiGroups: ["", "config.openshift.io"]
  resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
  verbs: ["get", "watch", "list"]

The k8sattributesprocessor requires permissions for pods and namespace resources.
The resourcedetectionprocessor requires permissions for infrastructures and status.

Bind the cluster role to the service account.

Example ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
subjects:
- kind: ServiceAccount
  name: otel-collector-deployment
  namespace: otel-collector-<example>
roleRef:
  kind: ClusterRole
  name: otel-collector
  apiGroup: rbac.authorization.k8s.io

Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the edge clusters.

Example OpenTelemetryCollector custom resource for the edge clusters

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: otel-collector-<example>
spec:
  mode: daemonset
  serviceAccount: otel-collector-deployment
  config:
    receivers:
      jaeger:
        protocols:
          grpc: {}
          thrift_binary: {}
          thrift_compact: {}
          thrift_http: {}
      opencensus:
      otlp:
        protocols:
          grpc: {}
          http: {}
      zipkin: {}
    processors:
      batch: {}
      k8sattributes: {}
      memory_limiter:
        check_interval: 1s
        limit_percentage: 50
        spike_limit_percentage: 30
      resourcedetection:
        detectors: [openshift]
    exporters:
      otlphttp:
        endpoint: https://observability-cluster.com:443 
        tls:
          insecure: false
          cert_file: /certs/server.crt
          key_file: /certs/server.key
          ca_file: /certs/ca.crt
    service:
      pipelines:
        traces:
          receivers: [jaeger, opencensus, otlp, zipkin]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [otlp]
  volumes:
    - name: otel-certs
      secret:
        name: otel-certs
  volumeMounts:
    - name: otel-certs
      mountPath: /certs

The Collector exporter is configured to export OTLP HTTP and points to the OpenTelemetry Collector from the central cluster.

Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the central cluster.

Example OpenTelemetryCollector custom resource for the central cluster

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otlp-receiver
  namespace: observability
spec:
  mode: "deployment"
  ingress:
    type: route
    route:
      termination: "passthrough"
  config:
    receivers:
      otlp:
        protocols:
          http:
            tls: 
              cert_file: /certs/server.crt
              key_file: /certs/server.key
              client_ca_file: /certs/ca.crt
    exporters:
      otlp:
        endpoint: "tempo-<simplest>-distributor:4317" 
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp]
  volumes:
    - name: otel-certs
      secret:
        name: otel-certs
  volumeMounts:
    - name: otel-certs
      mountPath: /certs

The Collector receiver requires the certificates listed in the first step.
The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, which in this example is "tempo-simplest-distributor:4317" and already created.