DŠ

The Observability Migration · 6 of 10

OTel Instrumentation

Wiring the OpenTelemetry SDK into real services: resource attributes, batch processors, and the settings that actually matter.

Instana to OTel: Same Concepts, Open Standard

At Instana, trace collection used proprietary agents and protocols. OpenTelemetry standardizes the same concepts: TracerProvider creates tracers, SpanProcessors handle batching, Exporters ship data over OTLP.

The difference: OTel is vendor-neutral. Instrument once, export anywhere. Change the endpoint URL, keep the instrumentation.

The Recipe

All our backend services run Django framework + MySQL + Redis + Kafka + AWS managed services. Same stack, same recipe:

otel.py, and every service gets one file:

def init_otel():
    endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")
    if not endpoint:
        return  # Skip in local dev

    resource = Resource.create({"service.name": os.environ["OTEL_SERVICE_NAME"]})
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(BatchSpanProcessor(
        OTLPSpanExporter(endpoint=endpoint, insecure=True)
    ))
    trace.set_tracer_provider(provider)

    DjangoInstrumentor().instrument()
    MySQLInstrumentor().instrument()
    RedisInstrumentor().instrument()
    RequestsInstrumentor().instrument()
    BotocoreInstrumentor().instrument()
    KafkaInstrumentor().instrument()

BatchSpanProcessor Tuning

The default BatchSpanProcessor works for most services, but high-throughput services need tuning:

BatchSpanProcessor(
    OTLPSpanExporter(endpoint=endpoint, insecure=True),
    max_queue_size=2048,
    max_export_batch_size=512,
    export_timeout_millis=30000,
)

max_queue_size=2048 - how many spans can be buffered before the processor starts dropping. The default (2048) is reasonable. For services handling thousands of requests/sec, increase to 4096 or 8192. If you see otel.bsp.spans_dropped in your metrics, this queue is full. max_export_batch_size=512 - how many spans per export RPC. The default (512) is fine. Larger batches are more efficient on the wire but add latency to the export cycle. Don’t set this larger than max_queue_size. export_timeout_millis=30000 - how long to wait for the collector to acknowledge an export before giving up. Default is 30s. If your collector is across a network boundary or under load, you may need to increase this. If it’s localhost or same-VPC, 10s is plenty.

The key thing: BatchSpanProcessor runs in a background thread and never blocks your application. If the queue fills up, it drops spans silently. Your application never slows down because of tracing, but you might lose telemetry under load if the queue is too small.

Resource Attributes

Beyond service.name, resource attributes are how you slice and filter telemetry across environments and versions:

resource = Resource.create({
    "service.name": os.environ["OTEL_SERVICE_NAME"],
    "deployment.environment": os.environ.get("OTEL_ENVIRONMENT", "unknown"),
    "service.version": os.environ.get("BUILD_VERSION", "dev"),
    "telemetry.sdk.language": "python",
})

In practice, we set most of these via the OTEL_RESOURCE_ATTRIBUTES environment variable rather than in code:

OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod-02,service.version=1.42.0

This keeps the code generic and the environment-specific values in deployment config (Kubernetes manifests, ECS task definitions, etc.).

These attributes appear on every span the service produces. In Grafana, deployment.environment powers the environment dropdown on every dashboard. service.version enables filtering traces by release, which is invaluable when debugging regressions (“did this start with the latest deploy?”).

Three Entry Points

Django starts differently depending on mode: gunicorn loads wsgi.py, runserver loads settings.py, management commands load manage.py. We call init_otel() from all three. It’s idempotent.

The Code Changes

Nearly 1:1 from the vendor agent to OTel:

# Vendor: @tracer.wrap() / span.set_tag()

# OTel:   @_tracer.start_as_current_span() / span.set_attribute()

Search-and-replace. The API surface is almost identical.

Lessons

Auto-instrumentation covers 90%. Django, MySQL, Redis, HTTP, AWS SDK are all instrumented automatically. Manual spans… Well, it’s a must on some places.

gRPC is noticeably faster than HTTP for OTLP export (binary protobuf vs JSON serialization). Worth switching early for high-throughput services.

Delete, don’t disable. Leftover vendor env vars create confusion. Leftover vendor packages create import conflicts. Clean cut.

Log-Trace Correlation

Trace IDs appear in log lines automatically when using the OTel logging bridge or when your logging format includes the trace context. In Python, the OTel SDK injects otelTraceID and otelSpanID into the logging context, so a standard JSON log formatter picks them up:

{"timestamp": "2026-04-14T12:00:00Z", "level": "ERROR", "message": "Device malfunction error", "otelTraceID": "abc123...", "otelSpanID": "def456..."}

In Grafana, this enables bidirectional navigation: click a log line with a trace ID, jump directly to the trace in Tempo. Click a trace span, see the correlated logs from that service during that span’s execution window.

This is the correlation that observability vendors charge premium pricing for, namely “unified traces and logs.” With OTel + Grafana, it’s configuration. The trace ID is already in the span. The same trace ID is already in the log line. Grafana’s datasource linking connects them. No proprietary agent, no vendor lock-in, no per-host pricing.

What Broke During Migration

Not everything was search-and-replace. Three issues required real debugging:

Django Channels (WebSockets): The standard DjangoInstrumentor covers WSGI, meaning synchronous HTTP request/response. It doesn’t cover ASGI WebSocket connections at all. WebSocket frames were invisible in traces. We had to use the ASGI-specific middleware from opentelemetry-instrumentation-asgi and wire it into the ASGI application directly, separate from the Django instrumentation. Kafka consumer initialization order: KafkaInstrumentor().instrument() patches the consumer class at import time. If you call it after the consumer is already instantiated and polling, the existing consumer instance isn’t patched. The fix was straightforward, call init_otel() early in the process lifecycle, before any consumer instances are created. But the symptom was confusing: some spans appeared (from new consumers) and some didn’t (from pre-existing ones), depending on import order. Startup race condition: One service had init_otel() called from a module that imported before the async event loop was ready. The gRPC exporter tried to establish a channel during import, failed silently (because error_mode on the exporter swallows connection failures), and every subsequent export was a no-op. No errors in logs, no spans in Tempo. We moved init_otel() to the ASGI lifespan handler where the event loop is guaranteed to be running.

The common thread: auto-instrumentation is magic until it isn’t. When it works, it’s invisible. When it doesn’t, the failure mode is silence: no errors, no crashes, just missing telemetry. Always verify each service produces spans after migration, don’t assume.

Next: Session 7. GitOps Deployment Model

All writing · Get in touch