On 17 November, after building a platform with no user apps on it, the cluster got its first real apps. A developer toolbox (IT-Tools), a notes app (Blinko) and a knowledge base (AFFiNE) moved off Docker Compose on the cloud host and onto K3s. The next day, a CouchDB for Obsidian sync followed.
The apps were the short part. Most of that day went into the six things the apps needed before the first one could start, and the guardrails that every later app inherits without asking.
This post covers the order I built things in, how each line of the Compose files became a Kubernetes object, what broke when the apps lost root and a writable filesystem, and how each move was checked before the old copy was switched off.
Why these apps, and why so few #
Project Aether started with 34 apps in Docker Compose inside one Proxmox LXC. One Docker restart took down all of them. ADR-010 moved most of them to an always-free Oracle Cloud VM, the app host, and kept the cluster for a handful of core apps. The control plane, kubelets and ArgoCD need their memory, and each worker has about 3.4 GiB to give.
So the first wave was chosen to cover the cases, not the count. IT-Tools is stateless: no database, no files, no egress. Blinko has a PostgreSQL database and a folder of files. AFFiNE adds Redis, pgvector, a migration step and about 68,000 rows of real data.
Platform first #
Before the first app, the cluster got the following, in this order, all on the same day:
| # | What | Why the apps need it | Covered in |
|---|---|---|---|
| 1 | Let’s Encrypt wildcard and *.lab names on the tailnet |
Real HTTPS on any device | Gateway API post |
| 2 | nfs-zfs StorageClass |
Files that survive losing a worker | Storage post |
| 3 | Shared PostgreSQL 16 (pgvector) and Redis | One database server, one role per app | ADR-022 |
| 4 | Nightly dumps, offsite copy, restore drill | Proof the data can come back | ADR-022 |
| 5 | Guardrails and a new-app template | Every app starts from the same safe shape | Below |
| 6 | Exporters, alerts and an outside HTTPS check | Know when it breaks | ADR-022 |
Backups came before data. The dump Job, the offsite copy and the restore drill were all in Git before a single real row moved in.
The guardrails every app inherits #
The apps namespace sets limits the API server enforces, so a forgotten resources: block can’t take memory the platform needs:
# LimitRange: a container that sets no requests or limits gets these, so nothing
# runs unbounded. ResourceQuota: a ceiling for the whole namespace, sized so the
# apps can never take the memory the platform on the same workers needs
kind: LimitRange
spec:
limits:
- type: Container
defaultRequest: { cpu: 25m, memory: 64Mi }
default: { cpu: 500m, memory: 256Mi }
---
kind: ResourceQuota
spec:
hard:
requests.memory: 2Gi
limits.memory: 4Gi
services.loadbalancers: "0" # apps are reached through the gateway only
services.nodeports: "0"Network access is denied by default, and egress is opt-in by pod label. An app asks for exactly what it uses:
# aether.dev/postgres: "true" → shared PostgreSQL (databases namespace, 5432)
# aether.dev/redis: "true" → shared Redis (6379)
# aether.dev/internet: "true" → public internet on 80/443 only; never the LAN,
# the tailnet or the clusterIT-Tools sets all three to "false", because every tool runs in the browser. Blinko asks for PostgreSQL and the internet (AI providers and link previews). AFFiNE asks for all three.
A shared Kustomize component adds the rest to every Deployment and StatefulSet: a priority class below the platform’s, no service-account token (apps don’t talk to the Kubernetes API), no injected service-link variables, and the runtime’s default seccomp profile. It deliberately doesn’t set runAsNonRoot. A patch can’t mean “only if unset”, so it would override an app that truly needs root. Each app sets that itself, and the template starts strict.
Compose to Kubernetes, line by line #
Here’s how Blinko’s and AFFiNE’s Compose files mapped onto the cluster. Every row is from the two versions in the repo:
| Compose | Kubernetes | What changed |
|---|---|---|
restart: unless-stopped |
Deployment, strategy: Recreate |
One writer to the files and the schema migrations, so no overlap during a rollout |
A postgres service per app |
A role and database on the shared PostgreSQL | AFFiNE’s old database used POSTGRES_HOST_AUTH_METHOD: trust. Now each app has its own password and can’t connect to another app’s database |
env_file: .env |
SOPS-encrypted Secret via envFrom |
Encrypted in Git, decrypted at sync (how) |
./data:/app/.blinko |
PVC on nfs-zfs |
Snapshotted hourly, copied offsite nightly |
depends_on: service_healthy |
startupProbe plus an ArgoCD sync hook |
The database is a platform service, and a sync hook creates each app’s role before the apps sync |
affine-migration with service_completed_successfully |
An initContainer |
Runs before every start, with the same image |
healthcheck: |
Startup, readiness and liveness probes | Three questions instead of one (see the takeaways) |
| Ran as root | runAsUser: 1000, read-only root, all capabilities dropped |
The part that broke things |
tsdproxy labels for the tailnet |
Service plus HTTPRoute |
One gateway and one certificate for every app |
Images were already pinned by tag and digest on the app host. Renovate now updates both sets of files, the Compose ones and the cluster’s.
AFFiNE’s migration step is the clearest example. In Compose it was a separate service the app waited on. On the cluster it’s an init container in the same pod, sharing the app’s environment through YAML anchors:
initContainers:
# Schema and data migrations before every start (the compose file's affine-migration).
- name: migrate
image: ghcr.io/toeverything/affine:0.27.4@sha256:b649f5ce2384ffdf13c23bccf81d759e15973d59d0b0058af65d895a84373099
command: ["node", "./scripts/self-host-predeploy.js"]
env: &env
- { name: HOME, value: /home/affine } # AFFiNE keeps ~/.affine/{config,storage}
- { name: REDIS_SERVER_HOST, value: redis.databases.svc.cluster.local }
envFrom: &envFrom
- secretRef: { name: affine } # DATABASE_URL, REDIS_SERVER_PASSWORD, AFFINE_PRIVATE_KEYWhat a read-only root broke #
All three apps ran as root with a writable filesystem on the app host. On the cluster they run as a non-root user with a read-only root filesystem, and each one found a different place it expected to write:
| App | What it tried to write | Fix |
|---|---|---|
| IT-Tools | The image’s entrypoint scripts rewrite /etc/nginx |
Skip them: nginx starts with its own config from a ConfigMap, on port 8080 as the nginx user |
| Blinko | npx and Prisma write caches under HOME |
HOME=/tmp, an emptyDir |
| AFFiNE | Its config directory sits under ~/.affine, a root-owned mount point |
Config gets its own emptyDir (6816219) |
| AFFiNE | The server writes a generated GraphQL schema to /app/src on start |
A small writable emptyDir at /app/src (42e5290) |
None of these was hard, but each one only showed up when the pod started. AFFiNE needed two follow-up commits after its first start. The fix is always the same shape: a narrow writable volume, with a comment saying why it exists.
The cutover, one app at a time #
Every stateful app followed the same steps:
- Deploy it on the cluster at 0 replicas, so the database role, the PVC, the route and the secrets all exist with nothing writing.
- Stop the app on the app host and move its data.
- Compare every table’s row count, and check the files.
- Scale to 1.
- Add an outside HTTPS check from Uptime Kuma on the DMZ observer.
- Keep the old copy stopped, not deleted, as the fallback.
| App | Data moved | Check |
|---|---|---|
| IT-Tools | None. First app from the template | Outside HTTPS check (fa74d57) |
| Blinko | Database and files | 23 tables, identical row counts, files restored (53c1847) |
| AFFiNE | Database (pgvector) and blobs | 91 tables, 67,852 rows identical, 156 blobs restored (0971155) |
| CouchDB (next day) | None: started fresh | The old copy held only system databases (ddc4cec) |
Memory came in where the requests say it would. IT-Tools used about 9 MiB on the app host and requests 16 Mi. Blinko idled at 139 MiB and requests 192 Mi. AFFiNE idled at 178 MiB and requests 256 Mi. All three fit well inside the 2 GiB quota.
The cutover had two side effects I didn’t plan for. That evening, a power cut outlasted the inverter and took the whole cluster down (part 12). The stopped copies on the app host were the fallback if the cluster’s data had been damaged. And the app host’s backup script kept trying to dump the two stopped databases, so it withheld its heartbeat (INC-2026-015). That one gets its own write-up.
What I learned #
- Build the restore before the data. The drill existed before real rows arrived, so the first check after the move used a known tool, not a new script. (ADR-022)
- Deploy at zero replicas first. Everything except the running process exists and can be checked before the data moves. (
979287c,5379efb) - A row count is a cheap, strong check. Identical per-table counts on both sides are a result I can point to, not a feeling that the move went fine.
- Moving an app changes the old host too. Anything there that still references the app (backup scripts, monitors) needs updating in the same change. (INC-2026-015)
- Read-only root finds every hidden write. Expect one or two per app, and document each writable volume. (
6816219,42e5290)
Architecture takeaways #
Why an init container replaces depends_on
#
Compose orders containers at start-up and then stops caring. Kubernetes has no start-order between pods, because pods are rescheduled, restarted and scaled independently, and an order that held at boot means nothing an hour later. What it offers instead is a guarantee inside one pod: init containers run to completion, in order, before the app container starts, on every start. A migration is exactly that kind of step, so it belongs in the pod. Dependencies between pods, like the database, are handled by retrying until ready, which is what the startup probe gives time for.
Why a namespace has its own resource policy #
Requests decide where a pod is scheduled, and limits cap what it can use. Leaving them out doesn’t mean “small”. It means BestEffort: the first pods evicted under pressure, and no cap on how much they take before that. A LimitRange gives defaults to anything that forgets, and a ResourceQuota caps the namespace as a whole. Together they turn “apps must not starve the platform” from a habit into something the API server rejects at admission.
Why there are three probes #
Compose’s healthcheck answers one question. Kubernetes asks three, because the answers lead to different actions. Startup: is it still booting? (Wait, don’t kill it during migrations.) Readiness: can it take traffic right now? (Remove it from the Service, don’t restart it.) Liveness: is it stuck? (Restart it.) Using one check for all three either kills slow starters or keeps sending traffic to a pod that can’t answer.
Previous: The Power Cut That Outlasted the Inverter · All posts in the series