Live status: status.chetan-thapliyal.cloud, a read-only public page with the platform’s live health, recovery objectives and incident history, served from the DMZ observer through an outbound-only tunnel.
Project Overview #
Project Aether is a personal platform engineering project. It started as a plan to move 34 Docker Compose services onto Kubernetes on one repurposed 2015 HP laptop (Intel i3, 12 GB of RAM, no public IPv4). The memory budget made that plan unrealistic, so the project became a platform split across three environments, all managed from one Git repository.
The aim is to practise the parts of platform engineering that do not depend on the hardware: declarative infrastructure, GitOps, least-privilege networking, secret handling, observability and recovery. The cloud compute runs on free tiers, so the platform costs very little to operate.
Architecture #
| Environment | Hardware | Role |
|---|---|---|
| Homelab | The laptop, running Proxmox VE with three Ubuntu VMs | K3s control plane and two workers, with ArgoCD delivering from Git |
| Oracle Cloud | Always Free ARM instance (2 vCPU, 12 GB) | Docker Compose applications, Kestra automation, central metrics and logs |
| Google Cloud | Always Free e2-micro, IPv6 only |
A low-trust observer that watches the other two from outside |
How the pieces fit together:
- Terraform provisions all three environments, with remote state in a GCS bucket. Ansible configures the hosts.
- ArgoCD watches the repository using the App-of-Apps pattern. A change pushed to Git is applied to the cluster within a few minutes.
- Tailscale is the network. Every machine carries a role tag, and access grants are explicit, so the observer can probe a few services and nothing more.
- SOPS encrypts secrets in Git to two keys, one offline age key and one cloud KMS key. Either can decrypt.
- Backups follow a 3-2-1 chain: nightly etcd snapshots, a copy on a second disk, and an encrypted, immutable offsite copy.
- Monitoring comes from two places. Prometheus, Grafana and Loki run on the app host, and an independent observer in another cloud raises the alert if the rest goes down.
Tech Stack #
| Layer | Technology | Purpose |
|---|---|---|
| Virtualisation | Proxmox VE 9.2 | VM isolation and snapshots on a single host |
| Kubernetes | K3s with embedded etcd | Lightweight cluster for the services that benefit from it |
| GitOps | ArgoCD, KSOPS | Declarative delivery and in-cluster secret decryption |
| Ingress | MetalLB, Envoy Gateway (Gateway API), cert-manager | LAN addresses, routing and internal TLS |
| Network | Tailscale, Tailscale Operator | Zero-trust access across all three environments |
| IaC | Terraform, Ansible, Task | Repeatable provisioning, configuration and operator commands |
| Secrets | SOPS, age, GCP Cloud KMS | Encrypted secrets in Git with two independent keys |
| Observability | Prometheus, Grafana, Loki, Grafana Alloy, Uptime Kuma | Metrics, logs and external checks |
| Automation | Kestra, n8n | Scheduled operations and webhook workflows |
| Backups | restic, Backblaze B2 | Nightly snapshots with an immutable offsite copy |
| CI | GitHub Actions, pre-commit | PR validation, drift checks, secret scanning |
Engineering Highlights #
- Decisions are written down. The repository holds 20 architecture decision records covering choices such as K3s over Talos, ArgoCD over Flux, and Tailscale grants over flat ACLs.
- Policy is tested. The tailnet access policy has automated tests that run on every change, so a mistake in a rule shows up before it is applied.
- Recovery is practised. Five disaster recovery scenarios are written as runbooks, and a cold restore from the offsite backup has been rehearsed.
- Failures are reviewed. Each real failure gets an incident review: what happened, the evidence, the fix, and the guardrail that prevents a repeat. There are twelve so far.
Challenges #
- A 12 GB budget. The original plan to run everything on the laptop did not fit in memory. Splitting workloads by blast radius across three environments gave a better design than the first one.
- An observer that could not see. The external monitor went blind to the app host for days, and the quick fix briefly gave the low-trust machine access to admin tools. The lasting fix was a tag model with one meaning per tag, plus policy tests.
- Backups that reported success. Every backup job exited cleanly, yet an audit found five different ways they were failing, from a heartbeat that never arrived to a “second disk” that was a folder on the first. Each is now a loud failure instead of a silent one.
- No high availability on one host. With a single physical machine, redundancy is not available. The design relies on fast, tested recovery: disposable VMs, a rebuild from Git and Terraform, and verified backups.
- Hardware limits. A worn fan and an unstable power supply caused noise and unexplained shutdowns. The hypervisor now reboots itself on a kernel panic, and alerts reach my phone from outside the machine.
Current Status #
- Foundation: complete. Proxmox, Terraform for all three environments, K3s, Tailscale, SOPS and the Oracle Cloud application stack.
- GitOps: complete. ArgoCD runs the App-of-Apps pattern with sync waves and encrypted secrets.
- Core platform: MetalLB, Envoy Gateway, cert-manager, the Tailscale Operator, default-deny network policies and log shipping are in Git. The etcd backup chain runs nightly.
- Application migration: not started on K3s. The applications still run on the Oracle Cloud host, and each will move only after its replacement has been proven.
Publishing #
The project is documented as a blog series, Project Aether: Building a Homelab:
- Project Aether: Building a Production-Grade Homelab on a Repurposed Laptop: the hardware, the architecture and the reasoning behind each major decision.
- Building My K3s Homelab: Creating a Proxmox Cloud-Init Template: one template that every node is cloned from.
- Homelab Hardware Mysteries: Fan Noise and Phantom Shutdowns: what the evidence showed, and one thing I got wrong.
- The Tag That Meant Two Things: a monitoring failure, a fix that widened access, and the tag model that ended it.
- Green Is Not a Status: five ways backups reported success while failing.
Key Learnings #
- Designing placement by blast radius: put each workload where its failure is cheapest.
- Running a monitor outside the failure domain it watches.
- Treating recovery as a feature that has to be tested, not assumed.
- Writing least-privilege network policy and testing it like code.
- Handling secrets in Git with two independent keys and no plaintext on disk.
- Writing incident reviews so each failure leaves a guardrail behind.