Every Kubernetes cluster that serves HTTP needs something at the edge: a component that accepts connections, terminates TLS and routes requests to the right Service. For years the answer was the Ingress API, usually with ingress-nginx behind it. That era is ending. The Ingress API is frozen, with new features going into Gateway API instead, and the community ingress-nginx project was retired in March 2026.
So when I built Project Aether’s K3s cluster, I skipped Ingress entirely. K3s ships with Traefik and its own load balancer, and I disabled both at install time. The edge is Gateway API, implemented by Envoy Gateway, with MetalLB giving it an address on the home network and cert-manager running an internal certificate authority.
The gateway was built, programmed and serving TLS for weeks before it had anything to route. That turned out to be useful: six things broke while building it, and two traps showed up before any traffic did. The first apps arrived on 17 November, and the last section covers what changed when they did.
Why not Ingress #
Ingress tried to describe everything in one resource: the listener, the TLS certificate and the routing rules. The spec covered only the basics, so every controller added its own features through annotations (nginx.ingress.kubernetes.io/rewrite-target, traefik.ingress.kubernetes.io/...). An Ingress written for one controller rarely works on another, and the annotations aren’t validated, so a typo just does nothing.
Gateway API splits the same job into typed resources, each owned by a different role:
| Resource | Who owns it | What it says |
|---|---|---|
GatewayClass |
Infrastructure provider | Which controller implements gateways of this class |
Gateway |
Cluster operator | Which ports listen, with which certificates, and which namespaces may attach routes |
HTTPRoute |
Application team | Which hostnames and paths go to which Services |
An application team can add or change routes without permission to touch ports or certificates. Features that used to be annotations, such as header matching, traffic splitting and redirects, are fields in the spec, validated by the API server.
K3s installs Traefik and ServiceLB (its built-in LoadBalancer implementation) by default. I didn’t want either running alongside the real edge, so the control-plane install disables both from the start:
sh -s - server \
--cluster-init \
--secrets-encryption \
--disable traefik \
--disable servicelbMetalLB: giving the gateway an address #
In a cloud, a Service of type LoadBalancer gets an address from the cloud provider. On bare metal with ServiceLB disabled, nothing allocates one, and the Service that Envoy Gateway creates would sit at EXTERNAL-IP <pending> forever.
MetalLB fills that gap. In L2 mode, it takes a pool of addresses on the home LAN and answers ARP requests for them: when a device asks “who has this address?”, a MetalLB speaker on one of the nodes replies “me”. If that node fails, another speaker takes over the claim.
The pool needed one piece of planning outside Kubernetes. MetalLB’s addresses must never be handed out by the router’s DHCP server too, or another device on the network could be given the gateway’s address. I shrank the router’s DHCP range and gave MetalLB a small block of six addresses outside it, with the first reserved for the gateway.
One detail surprised me. Envoy Gateway creates its Service with externalTrafficPolicy: Local, and MetalLB only announces an address from a node that has a ready endpoint for it. So the gateway address follows the Envoy proxy pod: whichever node runs the proxy answers for the VIP. A side effect of Local is that Envoy sees real client source addresses, which matters later for access logs and rate limiting.
The Gateway API objects #
Envoy Gateway is installed by ArgoCD from its Helm chart (gateway-helm, v1.9.2). The class and the gateway are plain manifests in Git:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: aether-proxy
namespace: gateway-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: aether-gateway
namespace: gateway-system
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
- name: https
protocol: HTTPS
port: 443
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: aether-wildcard-cert
allowedRoutes:
namespaces:
from: AllBoth listeners report Accepted, ResolvedRefs and Programmed as True. The gateway has an address from MetalLB and the HTTPS listener presents a certificate. For its first few weeks, zero HTTPRoutes were attached.
TLS: an internal CA with cert-manager #
The HTTPS listener needs a certificate. For a first version, cert-manager runs a small internal CA:
ClusterIssuer selfsigned-root-issuer (selfSigned)
└── Certificate aether-root-ca (isCA: true, in namespace cert-manager)
└── ClusterIssuer aether-ca-issuer (ca: aether-root-ca-secret)
└── Certificate aether-wildcard-cert (wildcard, in gateway-system)
└── Gateway listener httpsA self-signed issuer can usefully sign exactly one thing, the CA. The CA issuer then signs everything else. The root CA’s Certificate lives in the cert-manager namespace for a reason that’s easy to miss: a ClusterIssuer of type ca reads its Secret from cert-manager’s own namespace. Put the root anywhere else and the issuer reports secret not found.
Six things that broke on the way #
| # | Symptom | Cause | Fix |
|---|---|---|---|
| 1 | Gateway API CRDs permanently OutOfSync, with a prune in either app able to delete CRDs the other needed |
A separate gateway-crds Application and the Envoy Gateway chart both managed the same CRDs |
One owner: the chart owns every Gateway API CRD |
| 2 | Sync failed with metadata.annotations: Too long |
Client-side apply stores the whole object in an annotation capped at 262,144 bytes. Envoy Gateway’s envoyproxies CRD is about 1.4 MB |
ServerSideApply=true on the app |
| 3 | Server-side apply never took effect | Replace=true was set alongside it, and ArgoCD gives Replace precedence |
Drop Replace, which is destructive anyway |
| 4 | cert-manager issuers failed: no such kind | cert-manager’s chart defaults to crds.enabled: false |
crds.enabled: true |
| 5 | The Envoy proxy requested 512 Mi of memory with no limit and could land on any node | The chart values shape the controller. The proxy is generated from an EnvoyProxy resource, and there wasn’t one, so it got upstream defaults |
An EnvoyProxy resource, referenced from the GatewayClass, that pins the proxy and sets limits |
| 6 | A recovery runbook’s kubectl get pods -n envoy-gateway-system returned nothing |
Envoy Gateway names the proxy envoy-<namespace>-<gateway>-<hash>. Mine is envoy-gateway-system-aether-gateway-…, which looks like upstream’s default namespace. The real namespace is gateway-system |
Fix the runbook, and read resource names carefully |
Number 1 is worth expanding, because it applies to any chart that bundles CRDs. Helm treats a chart’s crds/ folder as one unit. Either the chart owns every CRD, or you manage all of them yourself and tell Helm to skip them. Splitting ownership between two Applications gives you two controllers arguing over the same objects forever.
Number 5 shows how much the defaults mattered here. The EnvoyProxy resource now pins the proxy to one worker and sets limits:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: aether-proxy
namespace: gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
pod:
nodeSelector:
kubernetes.io/hostname: k3s-worker-01 # data plane apart from the controller-heavy worker-02
container:
resources:
requests:
cpu: 50m
memory: 96Mi # measured 73 Mi idle (10 Nov)
limits:
cpu: 250m
memory: 256Mi # re-size after Phase 04 trafficThe proxy idles at 73 Mi, against a default request of 512 Mi. On a 12 GB laptop, that difference is a whole small service. The comment on the limit is deliberate: an idle proxy says nothing about a busy one, so the numbers get measured again once real traffic arrives.
Two traps found before any traffic #
The root CA would have re-keyed itself on Christmas Day #
Neither Certificate set a lifetime, so both got cert-manager’s defaults: 90 days, and, since cert-manager 1.18, a private-key rotation policy of Always. That’s right for a leaf certificate. For a root CA it’s a time bomb. On its first renewal, 25 December 2026, the root would have been reissued with a new key, and every device that trusted the old root would reject everything signed by the new one.
Nothing trusted the root yet, so nothing broke. But the moment I installed it on a laptop or phone, the clock would have started. The root now has a 10-year lifetime and keeps its key on renewal; the wildcard leaf keeps the defaults, because leaves should rotate (ADR-021):
duration: 87600h # 10 years
privateKey:
rotationPolicy: NeverThe same review found a second problem with the certificate’s names. The wildcard covers *.aether.local, and .local is reserved for multicast DNS (RFC 6762). macOS, iOS and many Linux systems send .local lookups to mDNS rather than to the router’s DNS, so the same name resolves differently on different devices. The fix was to use a subdomain of a domain I own, with publicly trusted certificates from Let’s Encrypt via the DNS-01 challenge through Cloudflare. DNS-01 needs no inbound access, and with a public CA there’s no root to install on every device. That’s now done, and *.aether.local is retired (next section).
The first route would have timed out #
The apps namespace has a default-deny NetworkPolicy. The first HTTPRoute would have been Accepted and Programmed, and every request through it would have timed out, because nothing allowed the Envoy proxy to reach pods in apps. That policy now exists, ahead of the first route, and it’s narrow: only the Envoy proxy pods in gateway-system may connect, not the controller or anything else in that namespace.
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
app.kubernetes.io/component: proxyThe namespaceSelector and podSelector sit in one from entry, so a connection must match both. As two separate entries they’d mean “either”, which would let every pod in gateway-system in, plus every Envoy-labelled pod in any namespace.
Then the first apps arrived #
On 17 November the gateway got its first real traffic: a developer toolbox, a notes app and a knowledge base moved onto the cluster. Three things changed at the edge to make that work (ADR-021 addendum).
Real names and real certificates. cert-manager now has a Let’s Encrypt ClusterIssuer that solves the DNS-01 challenge through Cloudflare, with an API token that can only edit DNS in that one zone. The HTTPS listener serves a single ECDSA wildcard for *.lab.chetan-thapliyal.cloud. Any device trusts it without installing anything, so the internal CA is left for internal needs only, which is what the 10-year root is for.
Reachable over the tailnet, from anywhere. The Tailscale operator publishes the Envoy data plane as one tailnet node, so my devices reach the gateway the same way at home and away. One public DNS record, *.lab, points at that node’s tailnet address. Anyone can resolve it, but only tailnet members can connect to it, so the rule became “no public records that point at anything reachable from the internet”. No split DNS, no resolver to run, no inbound port.
Routes, owned by the apps. Each app carries its own HTTPRoute next to its Deployment, attached to the shared gateway’s HTTPS listener. This is the role split from the start of the post, working as intended: the app says which hostname goes to which Service, and never touches ports or certificates.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: it-tools
spec:
parentRefs:
- name: aether-gateway
namespace: gateway-system
sectionName: https # HTTP is redirected for every lab name (gateway/http-redirect.yaml)
hostnames:
- it-tools.lab.chetan-thapliyal.cloud
rules:
- backendRefs:
- { name: it-tools, port: 80 }Two smaller routes belong to the platform. One redirects plain HTTP to HTTPS for every lab name. The other, ping.lab…, is answered by Envoy itself with a fixed response and no backend pod. A single request to it tests DNS, the tailnet path, the certificate and the gateway together, so it’s what the outside monitor checks. The NetworkPolicy from the previous section was already in place when the first route attached.
What I learned #
- Every CRD needs exactly one owner. Two Applications managing the same CRDs fight forever.
- Large CRDs need server-side apply, and
Replace=truesilently cancels it. - Chart values shape the controller, not always the workload it generates. Check what the generated pods actually request.
- A root CA is not a leaf. Defaults that are right for leaves (short lifetime, new key on every renewal) are dangerous for a root.
- Don’t use
.localfor unicast DNS. - Pre-position the network policy before the first route, or the first deploy fails in a way that looks like an application bug.
Architecture takeaways #
Why Gateway API replaced Ingress #
Ingress put three roles’ concerns into one object, and left everything beyond the basics to vendor annotations. Gateway API makes the split explicit: the infrastructure provider owns the class, the operator owns listeners and certificates, and application teams own routes. Each role can change its part without permission over the others, and features are typed fields the API server validates, not strings it ignores.
Why LoadBalancer needs a controller off-cloud
#
A Service of type LoadBalancer is a request, not a mechanism. In a cloud, the cloud controller manager fulfils it by creating a real load balancer. On bare metal, something else has to own a pool of addresses and make the network deliver traffic for them, which is MetalLB’s job here. Kubernetes leaves that to a pluggable controller on purpose: how an address reaches a node is a property of the network, which the cluster doesn’t control.
Previously in the series: Seven Silent GitOps Gotchas, and what ArgoCD and Helm didn’t tell me. Every part is listed on the series page.