OpenTelemetry — Implementation Guide for Metrics, Logs

Overview

OpenTelemetry (OTel) is a vendor-neutral observability framework for instrumenting, generating, collecting, and exporting telemetry data (traces, metrics, and logs). Maintained by the CNCF, it unifies what was previously fragmented across OpenTracing, OpenCensus, and vendor-specific agents. With OpenTelemetry, you instrument your application once and send data to any backend: Jaeger, Zipkin, Prometheus, Grafana, Datadog, New Relic, or cloud-native solutions.

When to Use

• For alternatives, see Observability — Metrics, Logs, and Traces Complete Guide.

• You want vendor-neutral instrumentation that outlives your current observability backend

• You need traces, metrics, and logs from the same application

• You are migrating between observability vendors and want to avoid re-instrumentation

• You operate polyglot environments (Go, Java, Python, Node.js, .NET)

• You need to collect telemetry from services you cannot modify (via the Collector)

Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│ Application │───→│  Collector  │───→│  Backend    │
│  (SDK)      │ OTLP│  (Agent/GW) │ OTLP│ (Jaeger/    │
└─────────────┘    └─────────────┘    │ Prometheus) │

Component	Role
SDK	In-app library that auto/manual instruments
Collector	Receives, processes, and exports telemetry
Exporter	Sends data to a specific backend
OTLP	OpenTelemetry Protocol (gRPC/HTTP)

Auto-Instrumentation (Python)

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

# Run your app with auto-instrumentation
OTEL_SERVICE_NAME=my-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317 \
OTEL_TRACES_EXPORTER=otlp \
OTEL_METRICS_EXPORTER=otlp \
OTEL_LOGS_EXPORTER=otlp \
opentelemetry-instrument python myapp.py

Manual Instrumentation

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

# Trace setup
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
span_exporter = OTLPSpanExporter(endpoint="collector:4317")

# Metric setup
metrics.set_meter_provider(MeterProvider())
meter = metrics.get_meter(__name__)
request_counter = meter.create_counter("http_requests_total")

# Use in code
with tracer.start_as_current_span("handle_request") as span:
    span.set_attribute("http.method", "GET")
    span.set_attribute("http.route", "/api/users")
    request_counter.add(1, {"method": "GET", "route": "/api/users"})
    # ... business logic

Collector Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

Context Propagation

OpenTelemetry propagates trace context across service boundaries using W3C Trace Context headers:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: vendor=value

This ensures a request traced in a frontend JavaScript app continues through Node.js, Python, and Go backends as a single trace.

Common Mistakes

• Forgetting to flush on shutdown — unflushed spans/metrics are lost when a pod terminates
• Exporting everything synchronously — always use batch processors to avoid blocking application threads
• No resource attributes — set service.name, service.version, deployment.environment on every signal
• Collector as a single point of failure — run Collectors as a DaemonSet or HA deployment
• Ignoring sampling configuration — default sampling may be too aggressive or too lenient; tune for your scale

Troubleshooting

• Pipeline fails silently: enable verbose logging and store pipeline artifacts between stages so you can inspect the exact state that failed.
• Container crashes on startup: check that environment variables, secrets, and config files are mounted correctly. Read the first 50 lines of logs before scaling replicas.
• Deployment rolls back repeatedly: verify health checks, resource limits, and startup probes. A failing readiness probe is a common cause of rolling restarts.
• Slow CI builds: cache dependencies and docker layers. Split large test suites into parallel jobs to reduce wall-clock time.
• Drift between environments: use infrastructure-as-code and immutable artifacts.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the opentelemetry and observability guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply opentelemetry — implementation guide for metrics, logs when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Advanced Topics

Scenario: OTel Instrumentation for E-commerce API

// Automatic instrumentation (Node.js)
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");
const { PrometheusExporter } = require("@opentelemetry/exporter-prometheus");

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: "http://otel-collector:4317"
  }),
  metricExporter: new PrometheusExporter({
    port: 9464
  }),
  instrumentations: [getNodeAutoInstrumentations({
    "@opentelemetry/instrumentation-fs": { enabled: false }
  })]
});
sdk.start();

// Manual instrumentation: custom spans
const { trace } = require("@opentelemetry/api");
const tracer = trace.getTracer("ecommerce-api");

async function checkout(cart) {
  return tracer.startActiveSpan("checkout", async (span) => {
    span.setAttribute("cart.items", cart.items.length);
    span.setAttribute("cart.total", cart.total);

    try {
      // Sub-span: validation
      const validation = await tracer.startActiveSpan("validate_cart",
        async (childSpan) => {
          childSpan.setAttribute("items.count", cart.items.length);
          const result = validateCart(cart);
          childSpan.setAttribute("valid", result.valid);
          childSpan.end();
          return result;
        });

      // Sub-span: payment
      const payment = await tracer.startActiveSpan("process_payment",
        async (childSpan) => {
          childSpan.setAttribute("payment.method", cart.paymentMethod);
          const result = await processPayment(cart);
          childSpan.setAttribute("payment.status", result.status);
          childSpan.end();
          return result;
        });

      span.setAttribute("checkout.success", true);
      span.setStatus({ code: 1 });
      return { validation, payment };
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: 2, message: error.message });
      span.setAttribute("checkout.success", false);
      throw error;
    } finally {
      span.end();
    }
  });
}

// Custom metrics
const { metrics } = require("@opentelemetry/api");
const meter = metrics.getMeter("ecommerce-api");

const checkoutCounter = meter.createCounter("checkouts.total", {
  description: "Total checkouts processed"
});
const checkoutDuration = meter.createHistogram("checkout.duration", {
  description: "Checkout duration in ms",
  unit: "ms"
});

// Usage:
checkoutCounter.add(1, { status: "success", method: "stripe" });
checkoutDuration.record(450, { status: "success" });

// Collector pipeline:
// App -> OTLP -> Collector -> Jaeger (traces)
//                          -> Prometheus (metrics)
//                          -> Loki (logs)

How do I migrate from Jaeger client to OpenTelemetry?

Replace the Jaeger SDK with the OpenTelemetry SDK. OTel exporters can send to Jaeger via OTLP. OTel auto-instrumentation replaces Jaeger manual instrumentation. Traces look identical in Jaeger UI. Migration is gradual: instrument new services with OTel first, migrate existing ones later.

End of document. Review and update quarterly.

Common Production Pitfalls

• Treating the guide as a checklist to complete once rather than a practice to evolve.
• Adopting every recommendation at once instead of starting with one measured change.
• Skipping the maturity assessment and forcing advanced practices on an unprepared team.
• Not updating runbooks and on-call expectations as new practices are introduced.
• Ignoring real incident data when prioritizing which parts of the guide to apply first.
• Failing to assign an owner who reviews decisions quarterly.
• Copying examples without adapting them to the team’s actual tooling and constraints.
• Forgetting to measure outcomes before adding the next improvement.