The Observability Migration · 1 of 10
OpenTelemetry Changed the Game
Why vendor-neutral instrumentation turned the monitoring backend from a marriage into a decision we can revisit.
The Lock-In Problem
Before OpenTelemetry, observability was a one-way door. Pick a vendor, adopt their agent, learn their query language, build dashboards in their format. Two years later you have hundreds of monitors, dozens of dashboards, and team muscle memory, all of it proprietary. Leaving means rebuilding from scratch.
OTel changed the equation. Vendor-neutral instrumentation. OTLP as the universal wire protocol. Instrument your code once, export to any compatible backend. If self-hosting doesn’t work, point the exporters at a SaaS. The instrumentation stays.
That’s the real game changer. OTel eliminates lock-in at the instrumentation layer. The backend becomes a choice you can revisit, not a marriage.
The Moment
It’s never the right time to replace something that works. There’s always a more urgent OKR. I wanted to do this 2 years ago but the priorities were elsewhere.
Then the monitoring contract renewal came up. The bills for logs was approaching the monitoring contract itself. That was enough to finally make the case.
The Fragmented Stack
Our observability was spread across multiple systems:
Logs → AWS CloudWatch. Functional but painful at scale: slow but expensive queries, and costs scale with both ingestion and query volume.
Traces → Commercial APM. Proprietary agent, proprietary protocol, proprietary query language. Everything locked in.
Metrics → Everywhere. Some in the APM vendor, some in CloudWatch, some in standalone Prometheus instances with no replication, no durable storage, no HA.
Three stacks, three query languages, three bills, no correlation between them.
The LGTM Stack
The Grafana LGTM stack (Loki, Grafana, Tempo, Mimir) is the open-source backend. Each component uses S3 as its primary object store:
- Loki: log chunks in S3, label-indexed
- Tempo: trace blocks in S3, trace-ID indexed
- Mimir: metric blocks in S3, PromQL-compatible
S3 gives you infinite retention, lifecycle tiering, and 11 nines of durability. Local disk is scratch space. The backend is durable by default.
My IBM Instana background, building APM internals, told us this architecture was sound. The concepts are the same: span collection, metric aggregation, time-series storage. Different implementations, open-source components instead of proprietary ones.
What We Built
EC2 instances across multiple AZs. Docker compose. Dedicated node pools per signal type. Different replication factors. NLB with health checks for failover. Terraform modules, config in Git, automated sync.
Predictable monthly cost. No overages.
But the real win isn’t cost: it’s that OTel made it possible to build without lock-in. If this stack doesn’t work in 3 years, we swap the backend. The instrumentation stays.
Coming Up
This is a 10-part series. Next: the architecture. Why EC2, why dedicated node pools, and how we designed for zero single points of failure.