Coherent Isolation: IndieAuth, Kubernetes, and the OSI Reality

The Starting Problem

I wanted comments on my Hakyll blog without Disqus, without Big Tech auth widgets, and without surrendering reader privacy. The decentralised path is IndieAuth – using your own domain as your identity – but wiring it into a static site requires a backend that handles the OAuth callback, stores the comment, and pushes JSON to the blog repository. I wrote rs-comment-api: a Rust microservice using warp, Turso libSQL, and git2, running as a NixOS systemd service on the host machine.

The service worked. The deployment around it did not, at least not consistently. This post is the accurate account of what actually broke, why, and what the fixes revealed about the topology of a virtualised Kubernetes cluster.

The Infrastructure

The cluster is two NixOS QEMU VMs (node1, node2) running Kubernetes 1.35 with CRI-O, Calico CNI in VXLAN mode, and etcd encrypted at rest via secretbox. node1 runs the control plane; both run kubelet and serve as GitLab CI runner hosts. Two Rust microservices are deployed in-cluster: ops-dashboard (webhook receiver and cluster status dashboard) and proxy-svc (stateless validation proxy). comment-api runs on the ThinkPad host, outside the cluster, because it requires git2 for repository operations and a stateful IndieAuth callback flow that cannot be split across replicas without shared session state.

The cluster domain is k8s.internal, not cluster.local. Every DNS reference in this post uses the actual domain.

What Actually Caused the 502

The public-facing 502 errors that opened this debugging session had a specific, narrow cause: comment-api.comment-api.svc.k8s.internal was not resolving inside the proxy-svc pods.

CoreDNS was crash-looping. The root cause was a forwarding loop: /etc/resolv.conf on node1 pointed at 127.0.0.53 (systemd-resolved), and the CoreDNS ConfigMap used forward . /etc/resolv.conf. CoreDNS forwarded queries to itself, detected the loop, and exited. Fix: change the forward plugin to explicit upstream resolvers (1.1.1.1, 8.8.8.8) rather than reading the host’s resolver config.

After CoreDNS recovered, comment-api.comment-api.svc.k8s.internal still had no backing Service or Endpoints object. The fix was a selector-less Service with a manual Endpoints pointing at 10.0.2.2:3000 – the QEMU NAT gateway address, which is how pods reach the host machine:

apiVersion: v1
kind: Endpoints
metadata:
  name: comment-api
  namespace: comment-api
subsets:
  - addresses:
      - ip: 10.0.2.2
    ports:
      - port: 3000

The Calico token framing in an earlier draft was wrong. Calico’s tokens were not the cause. The cause was DNS and a missing Endpoints object.

The Actual Dramatic Event: etcd Key Rotation

The more significant incident, omitted from earlier drafts, was this: the generate-keys.sh script regenerated EncryptionConfiguration.yaml.age on every invocation. Every run produced a fresh secretbox key. The new key was deployed to kube-apiserver via agenix; etcd still held secrets encrypted with the old key. Result: output array was not large enough for encryption on every secret read, cluster-wide – which cascaded directly into pods across every namespace failing to start with CreateContainerConfigError, since none of them could pull the secrets their containers depended on.

Recovery required adding identity: {} as the first provider in EncryptionConfiguration, restarting kube-apiserver, rewriting every secret with:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

then removing the identity provider and restarting again. This happened three times before the root cause was identified.

The fix was a single existence guard:

if [ ! -f "EncryptionConfiguration.yaml.age" ]; then
  # generate
fi

Rotating the secretbox key on a live cluster is a destructive operation that must be performed manually with a documented recovery procedure. It must not happen automatically.

Two Non-Obvious apiserver Defaults

A hand-rolled kube-apiserver has two defaults that block Calico entirely and are not obvious from the documentation:

--allow-privileged defaults to false. Calico requires privileged pods to program nftables rules and manage network interfaces. Without this flag, every calico-node pod fails at creation with Privileged container is not allowed.

PodSecurity admission is compiled in as a default admission plugin in Kubernetes 1.25 and later. Without explicitly disabling it, it blocks the Calico DaemonSet because calico-node violates the baseline security profile. The flags that fixed this:

--enable-admission-plugins=NodeRestriction
--disable-admission-plugins=PodSecurity
--allow-privileged=true

Calico on Kernel 6.12

The surviving configuration is codified as an idempotent patch app (deploy-calico-patches) rather than a one-time manual fix, since a Calico Helm upgrade or DaemonSet recreation would otherwise silently revert it. Three patches are reapplied on every run: FelixConfiguration forced to iptablesBackend: Auto rather than letting Felix autodetect (autodetection is what triggered the nftables/nft-segfault path in the first place); the default IP pool forced to vxlanMode: Always / ipipMode: Never, since IPIP encapsulation doesn’t traverse the QEMU virtio-net topology cleanly; and the calico-node readiness probe replaced with a direct calico-node -felix-ready exec check, because the stock HTTP readiness probe was reporting ready before Felix had actually finished programming routes on a freshly restarted node.

The OSI Map

Every debugging session reduced to the same question: which layer is this actually a problem on? The infrastructure maps cleanly:

• Layer 1 (Physical): Wi-Fi, QEMU virtio-net NICs, vde_switch connecting VMs.
• Layer 2 (Data Link): brkube bridge, veth pairs, vxlan.calico encapsulation.
• Layer 3 (Network): Node subnets (10.88.1.0/24), Calico pod IPs (172.16.x.x), wgkube WireGuard overlay, tailscale0.
• Layer 4 (Transport): TCP ports, Tailscale Funnel, kube-proxy DNAT rules.
• Layer 5/6 (Session/Presentation): Tailscale TLS termination, connection persistence.
• Layer 7 (Application): proxy-svc JSON validation, ops-dashboard webhook handling, Calico L7 policy.

The 127.0.0.1 problem appears in at least four forms in this cluster: on the host it resolves to the ThinkPad; inside node1’s shell it resolves to node1; inside a Docker container it resolves to the container; inside a pod it resolves to the pod. Every configuration file that hardcodes 127.0.0.1 is a future debugging session waiting to happen.

etcd Operational Housekeeping

The WAL grew to 2.1GB over the course of normal cluster operation, causing apply request took too long: 3.8s from etcd, which cascaded into kube-scheduler and kube-controller-manager losing their leader leases and restarting. The fix was automatic compaction and defragmentation on a systemd timer:

etcdctl --endpoints=unix:///run/etcd/grpc compact \
  "$(etcdctl endpoint status --write-out=json | jq -r '.[0].Status.header.revision')"
etcdctl --endpoints=unix:///run/etcd/grpc defrag

Running this on a timer prevents WAL accumulation from becoming a reliability event.

proxy-svc as a Typed Validation Boundary

proxy-svc sits between the public internet (via Tailscale Funnel) and comment-api. Tailscale Funnel is a door – it forwards public traffic directly into the cluster with no inspection. proxy-svc is the bouncer.

The validation pipeline uses typestate: each processing step is a type transformation, so the compiler enforces that forwarding cannot happen without passing through all validation stages.

Raw -> Shaped -> RatePassed -> Clean -> Forwarded

forward(req: Forwarded) cannot be called with Raw. The compiler makes it unrepresentable. This is not a performance optimisation – it is a correctness guarantee. The rate limiter (50 concurrent permits via tokio::sync::Semaphore) and the honeypot field check both participate in this chain. A bot that fills the honeypot field is rejected at the Clean stage; a request that exceeds 64KB is rejected at the Shaped stage. The kube-apiserver never sees either.

The Two-Node Reality

Earlier drafts and the README describe a single-node mental model. node2 exists and created its own class of problems: node2’s kubeconfig pointed at 127.0.0.1:6443 (its own localhost), not node1’s apiserver at 192.168.0.1:6443. The TLS certificate for kube-apiserver is valid for 10.88.0.1 and 10.0.2.15, not 192.168.0.1, which meant fixing the server address also required adding insecure-skip-tls-verify: true to node2’s kubeconfig. The certificate SAN includes config.kube.hostIp4, which is node1’s VDE address – adding the internal hostnames to the SAN at generation time would have avoided this entirely.

What the Infrastructure Actually Does

The cluster hosts two things and does them consistently:

GitLab CI runners execute Nix-based builds for the blog and the microservices. The CI image is a deterministic OCI image built with pkgs.dockerTools.buildLayeredImage containing Nix, Git, Attic, and the build toolchain. It is pushed to the in-cluster registry via skopeo over SSH and loaded into CRI-O before the first job runs. This means the runner environment is identical across every job invocation – no runtime patching, no missing binaries, no Cachix auth failures from a stale token in a mutable container.

Concretely, each service image (ops-dashboard, proxy-svc, echo-svc) is built reproducibly via crane/Nix from its own Cargo source tree, then pushed with the same three-step flow: nix copy ships the built image over SSH to the Tailscale-reachable build/deploy host, then skopeo copy loads it from the local docker-archive: path directly into the in-cluster registry at 10.88.1.1:5000 with TLS verification disabled (it’s an internal-only registry, not internet-facing). No image is ever built inside the cluster itself, and no image is ever pulled from a public registry – the entire supply chain from source to running container stays inside the VDE/Tailscale-bounded network.

The deployment target is the cluster itself. A GitLab pipeline success event sends a webhook to ops-dashboard, which patches the relevant Deployment’s restart annotation via the kube-apiserver API. The new pod pulls the updated image from the local registry. The entire path from git push to running pod traverses:

GitLab -> Cloudflare Tunnel -> ops-dashboard -> kube-apiserver -> kubelet -> CRI-O -> local registry

No external container registries, no cloud pull rates.

Getting the CI Image Onto the Runner Node

The service images (ops-dashboard, proxy-svc, echo-svc) go through the registry – built, pushed via skopeo to 10.88.1.1:5000, pulled by CRI-O on demand. The CI runner’s own image does not. It is streamed directly onto node1 with:

sudo docker save localhost/blog-ci:latest \
  | zstd -T0 \
  | ssh xameer@node1-1 "zstd -d | docker load"

The distinction matters and is not just a style choice. blog-ci is the image the GitLab runner itself executes inside to run every CI job – it has to be present and loadable before the runner can do anything, including before it could reach out to a registry to pull itself. Pushing it through the same registry-and-CRI-O path used for the service images would make runner bootstrapping depend on the in-cluster registry being reachable and healthy at exactly the moment nothing else in the pipeline has been verified yet – a circular dependency the service images don’t have, since something else (the running cluster) always exists first to pull them.

docker save linearises the full image (Nix store closure and all) to a tarball on stdout; zstd -T0 compresses it using all available CPU cores in parallel rather than the single-threaded default, which matters here because the image contains a full Nix toolchain closure – nix, git, cachix, attic-client, openssh, gnumake, plus their transitive store dependencies – and is large enough that single-threaded compression would meaningfully slow the push. The ssh ... "zstd -d | docker load" on the receiving end decompresses the stream and loads it directly into node1’s local Docker/CRI-O image store over the same SSH pipe, with no intermediate file written to disk on either end and no registry round-trip at all.

The image itself is built as a Nix-layered image (pkgs.dockerTools.buildLayeredImage) specifically so that this transfer stays cheap on repeated runs – Nix’s content-addressed store means unchanged layers (the base toolchain) don’t get re-transferred just because one dependency version bumped, only the layers that actually changed do. The environment baked into the image is deliberately narrow and explicit rather than inherited from the host:

Env = [
  "NIX_REMOTE=local"
  "USER=root"
  "SSL_CERT_FILE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"
  "NIX_SSL_CERT_FILE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"
  "GIT_SSL_CAINFO=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"
  "CURL_CA_BUNDLE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"
  "PATH=${pkgs.lib.makeBinPath [ ... ]}:/usr/bin:/bin"
];

Four separate CA bundle environment variables, all pointing at the same cacert derivation, exist because git, curl, nix, and OpenSSL-based tools each read a different variable name for their trust store by convention (GIT_SSL_CAINFO, CURL_CA_BUNDLE, NIX_SSL_CERT_FILE, SSL_CERT_FILE) – there is no single environment variable that all four tools respect, so all four are set explicitly rather than relying on any one of them to be inherited or defaulted correctly inside a minimal container that has none of the host’s usual certificate discovery paths. NIX_REMOTE=local and USER=root exist because the container has no multi-user Nix daemon running inside it – Nix has to operate directly against the local store as root, not delegate to a daemon socket that doesn’t exist in this image.

Conclusion

The 502 was a DNS failure and a missing Endpoints object. The encryption incident was a missing existence guard in a key generation script. The Calico failures were undocumented apiserver defaults and a kernel compatibility bug in a specific version. None of these were architectural failures. All of them were precise, specific, fixable problems that the debugging methodology described in the OSI map reduced to a single layer at a time.

The infrastructure is now stable. The generation script has the guard. CoreDNS has explicit upstream resolvers. The Endpoints object exists. The apiserver flags are documented. etcd is defragmented on a timer.

Copyright (C) 2013-2026 Xameer . All rights reserved.