The previous post covered how Project Aether’s cluster builds itself from one root Application. This one covers what went wrong after it worked.
None of these failures was loud. No pod crash-looped and no page went down. Each time, something reported success, or simply said nothing, while the system did less than I thought. A Helm setting was accepted and ignored. Four apps read Healthy for eight days while ArgoCD had stopped managing them. A webhook made pushes faster and quietly left five apps unable to compare with Git.
Seven gotchas, each with its evidence and the check that catches it now. They’re mostly about ArgoCD and Helm, but the pattern applies to any reconciler: the tool’s own status is a claim, not a fact.
1. Helm accepts settings that don’t exist #
My ArgoCD values file said applicationSet.enabled: false. The ApplicationSet controller kept running anyway.
The chart I was on (argo-cd 7.8.26) has no applicationSet.enabled key at all. Dex and Notifications are switched off with .enabled, so I assumed ApplicationSet was too, but its template has no such condition. Helm doesn’t validate values against the chart, so it accepted the key, stored it, and ignored it. The proof was in Helm’s own view of the release: helm get values -a didn’t list applicationSet.enabled at all.
helm pull argo/argo-cd --version 7.8.26 --untar --untardir argo-cd-chart
python3 -c "import yaml; d=yaml.safe_load(open('argo-cd-chart/argo-cd/values.yaml')); print('enabled' in d['applicationSet'])" # FalseThe fix was applicationSet.replicas: 0, a key the template does use. The cost here was small, about 20 MiB for a pod I didn’t need. The same pattern bit twice more, and those cost more:
- The Tailscale operator had
nodeSelectorandresourcesat the chart’s root, where the chart doesn’t read them (they belong underoperatorConfig). The operator ran on the wrong node with no limits, and the corrected values then got stuck behind gotcha 2. - Promtail had a
dnsPolicythe chart doesn’t support. Two of three nodes resolved Loki through public DNS and shipped their logs nowhere for days (INC-2026-006).
Three bugs, one cause. So the fix is one check for the whole class: a script that compares every Helm values file in the repo against the pinned chart’s own values.yaml, and fails on any key the chart doesn’t define. It runs in CI on every push. Run against history, it flags all three.
"""Fail when a Helm values key is not defined by the target chart.
Helm silently ignores unknown keys. That one behaviour caused Bug G (Phase 02),
Bug N and Bug X (Phase 03). This lints every Helm-sourced ArgoCD Application in
kubernetes/cluster/ plus the ArgoCD bootstrap values against the chart's own
values.yaml for the pinned version.
"""2. Healthy next to Unknown reads as “fine”
#
An audit on 2 November listed every ArgoCD app and found four showing Healthy and Sync: Unknown. People look at the first word. The second one meant ArgoCD could no longer compare those apps with Git, so it had stopped auto-syncing and self-healing them (INC-2026-005).
For eight days, the certificate manager, the gateway controller, the Tailscale operator and a metrics exporter were deployed but unmanaged. A change in Git did nothing, and a manual edit in the cluster would have stayed. One real change was lost: a fix moving the Tailscale operator to its intended node with resource limits was merged on 28 October and never applied.
Each app’s status explained it:
.status.terminatingReplicas: field not declared in schemaHealthy describes the workload. Synced describes ArgoCD’s grip on it. A GitOps tool that can’t build a diff will still report the workload as healthy, because the workload is. Since then, every sign-off check counts Unknown as a failure, not a footnote.
3. Upgrading the cluster can break a tool that compiled in its schema #
Why couldn’t ArgoCD diff those four apps? Kubernetes 1.33 added .status.terminatingReplicas to Deployments, and the cluster (1.36) fills it in. For apps that use server-side apply, ArgoCD 2.14 built its diff against a Kubernetes schema compiled into its own binary, from libraries older than that field. It met a field it had never heard of and gave up.
The cluster upgrade itself went fine. The breakage was in a tool that carried its own copy of the API’s shape. ArgoCD 3.5.7 knows the newer schema, and I’ve since upgraded. The durable guardrail is a review condition in the GitOps controller’s decision record: whenever the cluster’s minor version changes, re-check the controller’s Kubernetes library version.
This applies well beyond ArgoCD. Anything that embeds a copy of the Kubernetes API types (controllers, operators, CLIs, policy engines) can fall behind the cluster it talks to.
4. Server-side apply isn’t free, so don’t turn it on everywhere #
The bug in gotcha 3 only hit apps using server-side apply, and I had turned it on for every Helm app by default. Server-side apply exists mainly for very large resources, such as CRDs too big for the last-applied-configuration annotation that client-side apply relies on. Most of my apps didn’t need it.
A table of which apps used server-side apply and which contained Deployments explained all four failures and all twelve successes at once. The fix followed from it:
| App | CRDs too big for client-side apply? | Fix |
|---|---|---|
| Tailscale operator, metrics exporter | No | Dropped server-side apply |
| cert-manager, Envoy Gateway | Yes | Kept server-side apply, added server-side diff |
Server-side diff asks the cluster for the schema instead of using the compiled one, which side-stepped the bug for the two apps that needed server-side apply. The lesson: every non-default option widens the set of bugs you can hit. Turn options on where they’re needed, not as a house style.
5. A metric isn’t an alert #
ArgoCD was already exporting app status to Prometheus. The argocd_app_info metric had shown four apps as Unknown for eight days. Nothing was watching it.
- alert: ArgoAppNotSynced
expr: argocd_app_info{sync_status!="Synced"} == 1
for: 30m
labels:
severity: warning
annotations:
summary: "ArgoCD app {{ $labels.name }} is {{ $labels.sync_status }}"
description: "Unknown = ArgoCD cannot diff the app: no auto-sync, no self-heal. Check .status.conditions."The 30-minute for matters. A sync takes a minute or two and briefly reads OutOfSync, and that shouldn’t page. Thirty minutes stuck is a real problem. There’s a matching alert for health that isn’t Healthy or Progressing.
Collecting a metric feels like monitoring. It isn’t, until something reads it and tells a person.
6. Push-based sync changes the load, not just the speed #
By default, ArgoCD polls Git every three minutes. I added a signed GitHub webhook so a push reconciles in about a second. The first test push worked perfectly. Then two pushes arrived 38 seconds apart, and five apps went Unknown (INC-2026-013).
On each push, ArgoCD’s webhook handler refreshes every Application that tracks the pushed repository and branch, unless the Application says which paths it depends on. Mine didn’t, so every push queued ten manifest builds at once, even for a change to documentation. The repo server builds two at a time (sized for a 1 GB memory budget), and two of the apps decrypt secrets through the KSOPS sidecar. Two bursts in a row pushed some builds past the controller’s 60-second deadline. The repo server cached the failures and paused retries, so the apps stayed Unknown after the queue had drained.
The clue was in the sidecar’s log:
Generating manifests with deadline -7s from nowA negative deadline means the request arrived after it had already expired: it was queued, not slow. A build run by hand inside the sidecar took 2 seconds, which confirmed it.
The fix is one annotation on each repo-sourced Application:
metadata:
annotations:
argocd.argoproj.io/manifest-generate-paths: .Now a push refreshes only the apps with a changed file under their own path. I verified it with a docs-only push: ArgoCD logged the event, refreshed nothing, and every app stayed Synced. The 3-minute poll is still there as a fallback if a webhook is lost.
The polling interval had been quietly protecting the repo server by spreading work out. Moving from polling to push changes the load pattern, so test with two events close together, not just one.
manifest-generate-paths.7. Count the containers #
ArgoCD’s repo server runs with the KSOPS sidecar that decrypts secrets (covered here). In kubectl get pods -n argocd, a healthy repo server shows 2/2. If it ever shows 1/1, the sidecar isn’t there, and every app with encrypted secrets will fail to render while every other app stays green.
argocd-repo-server-79bd8c4b75-795z5 2/2 Running ← 2/2 = sidecar activeA dashboard that’s mostly green hides that kind of partial failure well. Two checks catch it: the container count after any change to ArgoCD, and a permanent smoke-test app whose only job is to decrypt a dummy Secret on every sync. If decryption breaks, that app goes red before anything real does.
Bonus: the number you measured once #
The decision record for ArgoCD set a ceiling of 1 GB. On 21 October, with a handful of apps, ArgoCD used 213 MiB, and that number went into my notes. With 16 apps it was 387 MiB, and the application controller was later OOMKilled at a 512 Mi limit. After the upgrade to 3.5.7, the total dropped from 434 to 310 MiB.
All of those were true when measured. Only the latest one is useful. Resource numbers belong next to the date and the load they were measured under, and they need re-measuring when either changes.
What I learned #
- Helm ignores unknown values. Lint values against the chart’s own
values.yaml, in CI. (INC-2026-006) Unknownis a failure.Healthysays the workload runs; onlySyncedsays GitOps still controls it. (INC-2026-005)- Upgrading the cluster means checking every tool with a compiled-in schema.
- Use non-default options where they’re needed, not everywhere.
- A metric does nothing until something alerts on it.
- Push-based sync changes load. Test bursts, and read the sign of a deadline. (INC-2026-013)
- Partial failures hide on green dashboards. Keep a canary for each critical path.
Architecture takeaways #
Why Kubernetes has multi-container pods #
KSOPS runs as a sidecar in the repo server’s pod rather than as its own Deployment. Containers in one pod share a network namespace and can share volumes, so ArgoCD talks to the plugin over a local socket, with no network hop, no Service and no extra authentication. The cost is gotcha 7: the pod’s overall status can hide that one of its containers is missing. That’s why the readiness column shows ready/total per pod, and why it’s worth reading.
A reconciler’s own state is a separate signal #
Kubernetes controllers report two different things: the state of what they manage, and whether they’re still managing it. ArgoCD splits these into health and sync. Most of the gotchas above live in the gap between them: a workload that’s healthy while its controller has lost the ability to compare, refresh or render it. Monitor both, and treat the controller’s ability to reconcile as something that fails like anything else.
Previously in the series: Building the GitOps Engine, ArgoCD’s app-of-apps and the sync waves that didn’t wait. Every part is listed on the series page.