DŠ

The Observability Migration · 7 of 10

GitOps Deployment Model

Terraform modules, configuration in Git and automated sync, so the observability stack can be rebuilt from scratch.

The Two-Phase Design

**gitops-sync.sh**runs on a systemd timer. Shallow git pull, hash comparison, Docker image pre-pull if changed, pending flag if config differs. Never restarts anything.

service-restart.sh is manually triggered. Checks the pending flag. Waits for peers to be healthy (up to 600s, which prevents cascading restarts). Then per-container: stop → 25s drain → start → wait for /ready (up to 120s).

The 25-second drain: NLB health checks every 10s with a 2-failure threshold. After docker stop, the NLB needs ~20s to detect the failure and drain connections. 5 seconds of buffer.

How Config Gets to Nodes

Each node has a git clone of the config repo at a fixed path. A systemd timer triggers the sync script every 60 seconds. The script does:

git fetch origin master --depth 1 && git reset --hard origin/master

Shallow clone, always tracks master. Any local modifications on the node are discarded (some AI agents like to do this…) if someone SSH’d in and hand-edited a config file (like previosuly mentioned AI agents), it gets overwritten on the next sync cycle. Nodes are cattle, not pets. The git repo is the single source of truth.

After pulling, the script hashes every config file and compares against the previous run. If anything changed, it pre-pulls any new Docker images (so the restart doesn’t block on a pull) and sets a pending flag. But it never restarts anything, that’s the controlled phase/instruction.

This means you can push a config change to master and know it will propagate to all nodes within 60 seconds, but nothing will act on it until an operator explicitly triggers the restart.

Per-Role Health Endpoints

The restart script checks different health endpoints depending on the service type:

Tempo → :3200/ready → Ingester ring membership, WAL replay complete
Prometheus → :9090/-/ready → Config loaded, targets scraped at least once
OTel Collector → :13133/ → Pipeline started, receivers listening
Loki → :3100/ready → Ring membership, WAL replay complete
Mimir → :9009/ready → Ingester/store-gateway ring membership, TSDB loaded
Grafana → :3000/api/health → Database connected, provisioning loaded

After starting a container, the script polls the corresponding endpoint with a 120-second timeout. Most services come up in 10-30 seconds. Loki and Tempo can take longer if they have large WALs to replay and this is normal after a node has been down for a while.

If the endpoint doesn’t return 200 within the timeout, the restart is aborted. The script exits with an error, and the operator investigates before touching other nodes. This prevents cascading failures because if a config change breaks one node, the other two stay healthy.

Why Never docker compose up -d

It restarts all changed containers at once. On a node running multiple services:

Multiple WAL replays competing for disk I/O and memory OOM kills during startup Incoming data dropped while collectors restart Alerts lost while alertmanager restarts

The correct approach: one container at a time, verify health between each.

Three Incidents Prevented

  1. YAML syntax error in Loki config. Synced to all 3 nodes. We tested restart on node 1 and Loki refused to start. Fixed the YAML, re-synced, restarted cleanly. Nodes 2 and 3 never saw the broken config applied.
  2. Invalid PromQL in a Prometheus rule. Applied to one node during restart. Prometheus caught it and refused to start. Fixed before touching other nodes.
  3. Docker image tag that didn’t exist. The sync script’s pre-pull step failed with a clear error. No restart attempted.

In all cases: damage limited to zero or one node.

A Typical Deploy

Push config to master. Wait ~60s for sync. SSH to node-1, run restart script. Check logs for errors. SSH to node-2, repeat. SSH to node-3, repeat. Check Grafana for data gaps.

Why Not Full Automation?

Failure modes are too varied. Sometimes you need a WAL wipe, sometimes a hotfix, sometimes the node itself has a problem. And the frequency is low enough that manual attention per deploy is cheaper than building and maintaining an automation framework.

Rollback Strategy

If a config change breaks a service, the rollback is simple: git revert the commit on master, wait for the sync cycle (or SSH to the node and trigger the sync script manually), then run the restart. Since the sync script always tracks master HEAD, reverting the commit is sufficient and the next sync pulls the reverted state.

For emergency cases where you can’t wait for git: SSH to the node and docker compose up -d with the previous image tag. All image tags are pinned in the docker-compose files (never:latest), so the previous working version is in git history. Check out the previous commit, restart the container, and you’re back.

The key insight: because config delivery (sync) and config activation (restart) are decoupled, a bad config that’s been synced but not yet applied is a non-event. Just revert and re-sync. Only configs that have been applied via restart need the emergency SSH path.

This two-phase model means most rollbacks are just git operations. The emergency path exists but is rarely needed.

Next: Session 8. Migrating Monitors

All writing · Get in touch