Metrics: Mimir Over Prometheus

The Prometheus Problem

Prometheus is excellent at scraping and short-term storage. But out of the box, it stores data locally with no replication. You can add Thanos or Cortex for long-term storage and HA, but that’s additional infrastructure and operational complexity.

For multi-environment observability with multi-year retention, we wanted something that handles durable storage and HA natively, without bolting on extra components.

Why Mimir

Mimir is PromQL-compatible but with S3 block storage. It runs all roles in one binary (scalable-single-binary):

distributor ingester querier store-gateway compactor ruler

Prometheus agents scrape targets and remote-write to Mimir. Multiple sources feed the same cluster.

Key Mimir Settings

multitenancy_enabled: false

limits:
  ingestion_rate: 150000
  max_global_series_per_user: 15000000
  max_label_names_per_series: 60

ingester:
  ring:
    replication_factor: 1

multitenancy_enabled: false selects single-tenant mode. All metrics from all environments go to one tenant. This simplifies queries (no X-Scope-OrgID header needed) and configuration. Multi-tenancy is useful when you need hard isolation between teams or customers; for a single organization’s observability, it’s overhead. replication_factor: 1 with HA tracker means we don’t replicate across ingesters. Instead, multiple Prometheus agents scrape the same targets and Mimir’s HA tracker deduplicates. This means write-path redundancy without doubling storage. For small clusters, RF=1 + HA tracker is simpler than RF=3 with quorum management. ingestion_rate: 150000 samples/sec is the per-tenant rate limit. Default is 10,000, which we hit within the first day. This limit prevents a misconfigured scrape (say, a loop creating millions of series) from swamping the cluster. Set it above your peak observed rate with headroom, not at your average. max_global_series_per_user: 15000000 is a hard cap on active series. Each active series consumes ~3-4KB in the ingester. 15M series is the ceiling before we’d want to investigate cardinality. This is a safety net, not a target. max_label_names_per_series: 60 prevents individual metrics from having too many labels. The default (30) was too low for some Kubernetes metrics that attach node, pod, container, namespace, and custom labels. 60 gives headroom without allowing unbounded label sets.

The Block Lifecycle

1. Sample arrives via remote-write
2. Distributor routes to correct ingester (by series hash)
3. Ingester holds in TSDB head (in-memory + WAL)
4. Head compaction every 2h → block shipped to S3
5. Compactor merges small S3 blocks into larger ones
6. Store-gateway syncs from S3 periodically
7. Querier reads recent from ingesters, historical from store-gateway

The question: where does “recent” end and “historical” begin? Two defaults control this:

• query_store_after (default 12h, ours: 0s): querier checks S3 for data older than this
• ignore_blocks_within (default 10h, ours: 0s): store-gateway skips blocks newer than this

With defaults, there’s a blind spot: data exists in S3 but nobody queries it there. After a migration, the new ingesters don’t have the old data, but the defaults say “trust them.” Result leads to 12-hour data gap.

We set both to 0s, always check both ingesters and store-gateway. More S3 reads, but zero gaps.

Cardinality Management

Cardinality is the number of unique time series in the system. Each active series costs ~3-4KB in the ingester. At 2M active series, that’s 6-8GB of memory just for series data. It grows silently and then hits a cliff. One day ingestion starts getting rejected with err-mimir-max-series-per-user.

We monitor cortex_ingester_active_series as a top-level health metric. When it trends upward unexpectedly, we investigate before it hits limits.

The biggest cardinality surprise came from Tempo’s metrics-generator. As covered in Session 4, an auto-instrumentor was creating thousands of unique span names per device endpoint. Tempo’s metrics-generator dutifully created a Prometheus series for each {service, span_name, status} combination, thousands of new series per day. The span-name normalization in the OTel Collector (the transform processor that collapses device-specific names into generic ones) was critical. Without it, we’d have blown through cardinality limits within a week.

Practical advice: set max_global_series_per_user conservatively and increase it deliberately. It’s much easier to raise a limit than to clean up after a cardinality explosion has already destabilized your ingesters.

HA Deduplication

Multiple Prometheus agents scrape overlapping targets for write-path redundancy. Mimir’s HA tracker deduplicates using cluster and __replica__ labels, keeping one sample per scrape. Redundant collection, not redundant storage.

Remote-Write Dependencies

Multiple config files across multiple repos all reference Mimir’s endpoint. When Mimir moves, you need to update ALL of them. The actual dependency list:

Prometheus agents on infra nodes (prometheus.yml → remote_write.url) scrape node_exporter, cAdvisor, and local service metrics. If this breaks, infrastructure dashboards go empty. Tempo metrics-generator (`tempo.yml` → metrics_generator.storage.remote_write) generates RED metrics (rate, errors, duration) from traces and writes them to Mimir. If this breaks, all tracing dashboards lose their metric panels: service maps, request rate, error rate and latency. The traces themselves still exist in Tempo, but the derived metrics that power Grafana panels disappear. EKS Prometheus agents (Helm values in the infrastructure repo) scrape Kubernetes cluster metrics and pod metrics via service discovery. If this breaks, application-level dashboards go empty.

We maintain this as a checklist. The one time we forgot entry #2 during a Mimir migration, tracing dashboards went flat for hours and we didn’t immediately connect “Mimir moved” with “trace dashboards are empty” because traces themselves were fine. It was the derived metrics that stopped flowing.

Store-Gateway Sync Lag

The store-gateway syncs block metadata from S3 on a configurable interval (ours: every 15 minutes). This means when an ingester flushes a 2-hour block to S3, the store-gateway doesn’t know about it for up to 15 minutes.

Combined with the query_store_after and ignore_blocks_within settings, this creates potential windows where data is invisible:

1. Ingester holds data in memory (TSDB head)
2. Head compaction produces a block, shipped to S3
3. Ingester may no longer serve this data (it’s been compacted out)
4. Store-gateway hasn’t synced yet, so it doesn’t know the block exists
5. Querier asks both, neither has the data → gap

Setting query_store_after=0s and ignore_blocks_within=0s closes this by ensuring the querier always asks the store-gateway, and the store-gateway always considers all blocks. The tradeoff is slightly more S3 API calls (the querier hits both paths on every query), but for a small cluster the cost is negligible compared to missing data.

If you’re running a larger cluster where this overhead matters, you can tune sync_interval down to 5 minutes instead of adjusting the query settings. The right answer depends on your query volume vs your tolerance for brief gaps.

---

Next: Session 6. OTel Instrumentation