aman®
← all writeups

Moving three clusters to GitOps

Replacing hand-run kubectl and a deploy script nobody wanted to own with ArgoCD and one repo of truth.

Date
2025-08-14
Type
build
Duration
7 weeks
Impact
140 applications migrated. Config drift went from weekly to zero in the following quarter.

PLACEHOLDER POST — seed data. Replace with your real writeup.

Before this project, “what is running in production” was answered by kubectl get and hope. Deploys went through a 900-line bash script with three authors, none of whom still worked on the team.

Where we started

  • 3 clusters (dev, staging, prod), each slightly different in ways nobody had written down
  • Helm releases installed from laptops
  • Hotfixes applied with kubectl edit, and never committed

What we built

One platform-apps repo using the app-of-apps pattern. Each environment is a directory of values overlays. ArgoCD ApplicationSets generate one Application per service per cluster.

generators:
  - matrix:
      generators:
        - clusters: { selector: { matchLabels: { tier: workload } } }
        - git: { directories: [{ path: "apps/*" }] }

How we migrated without breaking anything

  1. Import each live release in manual sync with self-heal off, and diff it
  2. Commit whatever the live cluster actually had, even when it was wrong
  3. Fix the drift in a separate PR, so every fix was reviewed
  4. Enable auto-sync one namespace at a time, starting with dev

What I’d do differently

Start with the drift report. Showing the team how far live state had wandered from the repo made the case for GitOps better than any design doc did.

← all writeups