↓ Skip to main content

Building the GitOps Engine: ArgoCD, App-of-Apps, and Sync Waves That Didn't Wait

2183 words·11 mins· loading · loading ·
Table of Contents
Project Aether – Building a Homelab - This article is part of a series.
Part 8: This Article

Nothing in Project Aether’s cluster is applied by hand except two things: a Helm release for ArgoCD and one Kubernetes Secret holding a decryption key. Everything else, from namespaces to cert-manager to the Tailscale operator, is created by ArgoCD from a single folder in Git. If I delete a Deployment with kubectl, it comes back. If I change one in the cluster, the change is reverted.

Getting there was mostly straightforward. Two things weren’t. My sync waves looked like they ordered the platform, but they only ordered creation, and later waves started while earlier ones were still coming up. And the one layer ArgoCD can’t manage, ArgoCD itself, quietly drifted from Git for three weeks, including a memory limit that got the controller OOMKilled.

This post covers the bootstrap, the app-of-apps layout, how the waves are actually assigned, and both of those problems with their fixes.

Delivery: a push to main runs the same checks as the local hooks in GitHub Actions; a signed webhook goes through Cloudflare and the edge VM's tunnel to ArgoCD over the mesh; ArgoCD refreshes only the apps whose paths changed, and the repo server with KSOPS fetches Git, builds manifests and decrypts secrets with the age key before applying them. Renovate opens weekly pull requests; Ansible and Terraform are run by hand. SOPS encrypts every secret for both a local age key and a Cloud KMS key.
The whole delivery path as it runs today, including the webhook and CI that came after this post's first version. This post is about the bottom half.

Why ArgoCD
#

The constraint was memory. The cluster shares a 12 GB laptop, and the worker that hosts platform services had roughly 3 GB to give. Flux v2 is lighter (around 150 MB), and I considered it seriously (ADR-009). I chose ArgoCD for the built-in UI: when a sync goes wrong, a resource tree with live health and a visual diff is the fastest way I know to see why. Flux needs add-ons for that.

The ADR set two fitness functions, and I hold the setup to them: ArgoCD’s components together stay under 1 GB steady state, and a commit to main reaches the cluster within three minutes. The second one came free at first, since ArgoCD polls Git every 180 seconds by default. Now a signed GitHub webhook tells ArgoCD about a push within about a second, and polling is the fallback. (Adding the webhook caused its own small incident, INC-2026-013: every push refreshed all ten apps at once, and five went Unknown until the refreshes were limited to apps whose folder changed.)


The bootstrap: the only imperative steps
#

ArgoCD can’t install itself, and it can’t decrypt secrets without a key it doesn’t have yet. So the bootstrap is two Taskfile tasks, run once per cluster:

  1. task k8s:sops-secret creates the argocd namespace and the Secret holding the age key that the KSOPS sidecar uses (the previous post covers why that one secret can’t come from Git).
  2. task k8s:bootstrap-argocd installs ArgoCD with Helm, from values that live in Git:
YAML
  k8s:bootstrap-argocd:
    deps: [k8s:sops-secret]
    cmds:
      - helm repo add argo https://argoproj.github.io/argo-helm
      - helm repo update argo
      - kubectl apply -f kubernetes/bootstrap/argocd/cmp-ksops-config.yaml
      - >-
        helm upgrade --install argocd argo/argo-cd
        --version {{.ARGOCD_CHART_VERSION}}
        --namespace argocd
        --values kubernetes/bootstrap/argocd/values.yaml
        --wait --timeout 5m --rollback-on-failure

The chart version is pinned in one variable at the top of the Taskfile, and --rollback-on-failure (Helm 4’s name for --atomic) puts the previous release back if the upgrade doesn’t come up healthy within five minutes.

Then one kubectl apply of the root Application, and from that point on I don’t touch the cluster with kubectl apply again.


App-of-apps: one root, many children
#

The root Application points ArgoCD at kubernetes/cluster/:

YAML
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-apps
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/ChetanThapliyal/Project-Aether.git
    targetRevision: main
    path: kubernetes/cluster
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true

Every file in that folder is itself an Application: one for cert-manager, one for MetalLB, one for Envoy Gateway, one for the Tailscale operator, and so on, fourteen in all. Some point at an upstream Helm chart with a pinned version. Others point back into the repo, at a folder of plain manifests. Adding a platform component is one new file in kubernetes/cluster/, and removing one is deleting that file.

Two settings turn Git from a suggestion into the source of truth:

  • selfHeal: true reverts any change made in the cluster that isn’t in Git.
  • prune: true deletes anything ArgoCD created that’s no longer in Git.

The finalizer matters too. Without it, deleting an Application leaves everything it created running, orphaned. With it, deleting a child’s file from Git takes the child’s resources with it.


Sync waves: the order on paper
#

Some things have to exist before others. A Certificate can’t be created before cert-manager’s CRDs exist, and nothing can go into a namespace before the namespace does. ArgoCD orders a sync with the argocd.argoproj.io/sync-wave annotation: lower waves first, and each wave waits for the previous one to be healthy.

This is how the waves are actually assigned in the repo today:

Wave What Why there
-3 Namespaces with their Pod Security labels Everything else lands in one
-2 cert-manager, MetalLB They install CRDs and controllers others depend on
-1 cert-manager issuers, MetalLB address pool, Envoy Gateway, the Tailscale operator’s secret Configuration that needs those CRDs, and shared secrets
0 Tailscale operator, the Gateway itself, Alloy, metrics exporters Platform services
1 NetworkPolicies, apps exposed on the tailnet Things that need the platform running

That’s the plan. For the first few weeks, it didn’t work the way the table suggests.


Problem 1: the waves ordered creation, not readiness
#

“Each wave waits for the previous one to be healthy” is true for ordinary resources, like a Deployment or a Service. It isn’t true, by default, for an Application inside an app-of-apps. ArgoCD stopped assessing the health of Application resources years ago, so to the root app, a child Application is “healthy” the moment it exists. Wave -2 would create the cert-manager Application, the root would see it as done, and wave -1 would start creating issuers while cert-manager’s pods were still pulling images.

On a running cluster that’s invisible, because everything already exists. On a cold rebuild, it’s exactly when ordering matters. I confirmed it by asking the root app what health it saw for its children:

Bash
kubectl get application -n argocd root-apps \
  -o jsonpath='{range .status.resources[?(@.kind=="Application")]}{.name}={.health.status}{"\n"}{end}'

Every child came back with an empty health value. The waves had nothing to wait on.

The fix is a custom health check in ArgoCD’s config, written in Lua, that reports a child Application’s own health back up to its parent:

YAML
    resource.customizations.health.argoproj.io_Application: |
      hs = {}
      hs.status = "Progressing"
      hs.message = ""
      if obj.status ~= nil and obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
      return hs

The nil guard matters. A brand-new child has no status yet, and the guard makes it report “Progressing” instead of failing on the missing field. After the change and a hard refresh of root-apps, every child reports a health value, and a wave waits for the previous wave’s children to be Healthy.

Reference: Application health in my knowledge base covers sync versus health status, the built-in checks, and how probes drive wave progression.

There’s a trade-off, and I took it on purpose: a Degraded child now blocks every later wave. For platform ordering that’s what I want. If cert-manager is broken, nothing that depends on certificates should try to deploy.

Two rows of the same ArgoCD sync: root-apps followed by waves -3, -2, -1, 0 and 1. In the first row, with no health check on child Applications, each wave starts as soon as the previous one exists. In the second row, a Lua health check reports each child's health to the root app, so each wave waits until the previous one is Healthy.
The same five waves, without and with the Application health check. Without it, the waves only order creation; with it, each wave waits for the one before to be Healthy.

Problem 2: the layer ArgoCD can’t manage drifted from Git
#

ArgoCD manages everything under kubernetes/cluster/. It doesn’t manage its own installation: that’s the Helm release from the bootstrap step. Nothing reconciles it. If I edited values.yaml and forgot to re-run the task, Git and the cluster would disagree, and nothing would tell me.

That’s what happened. An audit in early November found the Helm release still at revision 2, from 21 October, while Git had moved on:

  • The live ArgoCD config still had an old placeholder URL instead of the one in Git.
  • The application controller was OOMKilled at its 512 Mi limit on 2 November. Its steady state with 16 apps was 229 Mi, so something pushed it to more than twice that.
  • The Application health check above wasn’t live, so the waves still didn’t wait.

All of those fixes were in Git or about to be. None of them were in the cluster.

The fix had two parts. First, apply Git: the controller limit went to 768 Mi, the health check went in, and the release moved to revision 3. Second, make the drift visible, with a task that compares the live release with Git and fails if they differ:

Python
live = yaml.safe_load(subprocess.check_output(["helm", "get", "values", "argocd", "-n", "argocd", "-o", "yaml"]))
git = yaml.safe_load(open("kubernetes/bootstrap/argocd/values.yaml"))
rel = json.loads(subprocess.check_output(["helm", "list", "-n", "argocd", "-f", "^argocd$", "-o", "json"]))[0]
problems = []
if live != git:
    problems.append("values differ from kubernetes/bootstrap/argocd/values.yaml")
if rel["chart"] != "argo-cd-{{.ARGOCD_CHART_VERSION}}":
    problems.append(f"chart is {rel['chart']}, Git pins argo-cd-{{.ARGOCD_CHART_VERSION}}")
if problems:
    sys.exit("DRIFT: " + "; ".join(problems) + ". Run: task k8s:bootstrap-argocd")
print(f"ok: {rel['chart']} rev {rel['revision']} matches Git")

I ran task argocd:drift before applying and got DRIFT, which proved the check could fail. After applying it said ok … rev 3. A bootstrap change now has a check on both sides of it.


Problem 3: “managed by ArgoCD” isn’t “tracked in Git”
#

The same audit found a quieter version of the drift problem: files the rebuild depends on that Git didn’t track at all. The central SOPS-encrypted secrets file for Ansible was in .gitignore until 24 October. The scripts that the pre-commit hooks call were ignored until 10 November, so a fresh clone had hooks pointing at files that didn’t exist. A cold rebuild from Git would have failed on files that only existed on my laptop.

Both are tracked now, and a pre-commit check fails if an Ansible playbook copies a repository file Git doesn’t track:

Bash
while IFS= read -r src; do
  path="$(realpath -m "$src")"
  if ! git ls-files --error-unmatch "$path" >/dev/null 2>&1; then
    echo "UNTRACKED: playbooks/$src"
    status=1
  fi
done < <(grep -hoE 'src:[[:space:]]*"?\.\./[^"[:space:]}]+' ./*.yml | sed -E 's/^src:[[:space:]]*"?//' | sort -u)

Fixed later: targetRevision: HEAD
#

Until 12 November, the root app and every repo-sourced child used targetRevision: HEAD. That doesn’t mean “whatever I just pushed”. It means the repository’s default branch. It was harmless only because I deploy from main and main is the default. If the default branch ever changed, or I pushed a fix to another branch expecting it to deploy, nothing would happen and nothing would say so. Every Git-sourced Application now names main explicitly (b9d8840), which is why the root app above says main.


What I learned
#

  • Sync waves wait on health, and by default a child Application has none. In an app-of-apps, add the Application health check, or the waves order creation only.
  • Whatever installs the GitOps tool is outside GitOps. Give that layer its own drift check, and prove the check fails before you trust it when it passes.
  • Memory limits need headroom for the spike, not the steady state. 229 Mi steady was OOMKilled at 512 Mi.
  • Everything a rebuild needs must be in Git, not just everything ArgoCD syncs. A file that only exists on one laptop is a single point of failure.
  • HEAD means the default branch. Name the branch you mean.

Architecture takeaways
#

Why Kubernetes reconciles instead of reacting
#

A CI pipeline is edge-triggered: a push happens, it runs a script, it exits. If someone deletes a Service an hour later, the pipeline never knows. Kubernetes controllers, and ArgoCD with them, are level-triggered: they repeatedly compare desired state with actual state and act on the difference, whatever caused it. That’s why selfHeal works, and it’s also why the bootstrap drift was possible: the Helm release has no controller comparing it to anything. The fix was to add the comparison, by hand, as a task.

Why ArgoCD’s state lives in CRDs
#

An Application is a Kubernetes object, stored in etcd and readable with kubectl, not a row in a separate database. That’s what makes app-of-apps possible at all: a parent Application can create, prune and health-check child Applications exactly as it would a Deployment. It’s also why the health fix was a Lua snippet in a ConfigMap, not a code change: ArgoCD treats its own resource type like any other.

Every system has an unmanaged root. Keep it small and checked
#

However declarative the system, something has to be installed by hand first. Here it’s one Helm release and one Secret. The goal isn’t to remove that root, which isn’t possible. It’s to keep it as small as possible, keep its inputs in Git, and give it the comparison that the rest of the system gets from its controllers.

Ordering needs health, not just sequence
#

A sequence that doesn’t wait for readiness is only a list. The waves became an ordering mechanism only when each step could report “I’m ready”, and the cost of that is accepting that a broken step stops everything after it. For a platform’s foundations, that’s the right failure mode.


Previously in the series: secrets in a public GitOps repo, with SOPS, age and GCP Cloud KMS. Every part is listed on the series page.

Project Aether – Building a Homelab - This article is part of a series.
Part 8: This Article

Related