Tale of a GitLab Job Pipeline

Architecture

• The VM

  • Deployment from host over tailscale ssh to remote vm TTY

When nixos-rebuild switch activates a new generation on node1 with

export NIX_SSHOPTS="-t"
nixos-rebuild-ng switch \
  --flake .#node1 \
  --target-host xameer@100.105.10.61 \
  --build-host xameer@100.105.10.61 \
  --use-remote-sudo \
  --ask-sudo-password \
  --impure \
  -j 4

systemd restarts services － including tailscaled itself if it or any of its dependencies changed. The moment tailscaled restarts, the SSH connection over Tailscale drops. The copy phase nix copy was mid-flight when this happened. leaving the store in a partially written state: paths recorded in the SQLite DB but missing from disk, with foreign key constraints preventing cleanup.

With some python the due sql ops for dangline store paths store-fix.py － automates the tasks to find all DB entries whose paths don’t exist on disk, delete their refs in both directions, then re-enable constraints. After that nix-collect-garbage and a fresh rebuild restored the system.

The runner token and attic token failures that followed were secondary － the store corruption had wiped the previous activation, so agenix secrets from the old generation were gone and services re-registered with stale or missing credentials.

The fix order that worked: repair store → rebuild → fix runner token newline → regenerate attic token against the live HS256 secret.

- Disk Pressure

As the VM does run a non trivial bunch of kubernetes api managed services, encrypted with etcd and deployed with kubectl. Some nix store paths’ closure fails to be copied to remote vm due to OOM erros, at times the gitlab-runner hosted by this vm [ which deploys this blog ] also fails for the same reason. To observe

 watch -n2 'free -h && echo "---" && ps aux  | head -5'
 #for suspecios units
 top -bn1 | head -20 | grep etcd
 # to observe
 df -h /home/xameer/clone/
 #direnv and devenv paths form kubectl deployments, which can also be done from gitlab CI with k8s executors, but so long as the runner is hosted here that too shall die unless fixed

Once top has shown the zombie systemd units or processes causing OOM just stop them with systemctl and clean the grabage and rebuild. - Binary Caching layer

The Problem – Deployment state

Nix captures the declared state – packages, service configurations, volume mounts, environment variables, and their dependencies – reproducibly and hermetically. What it cannot capture is operational state for devops Arion

Traditional CI/CD often results in: [Declared Nix] -> [Imperative Runner Setup] -> [Imperative Deploy Script] -> [Operational State] Hercules CI/Arion results in: [Declared Nix + Hercules Agent Config] -> [Hermetic Build] -> [Declarative Effect/Arion] -> [Operational State] They make the operational state, such as which runner handles a job or how containers are structured, part of the declared, version-controlled Nix configuration, thus moving it closer to the ideal of perfect reproducibility.

Terraform and Kubernetes do not reduce this gap – they shift it. Terraform manages cloud resource declarations, but still requires manual secret rotation, manual runner registration, and manual responses to OOM events, unless paired with auto-scaling and pod disruption budgets. Kubernetes adds liveness probes, resource limits, and automatic pod restarts, which would have caught the etcd memory balloon automatically via resources.limits.memoryand restarted the pod before it consumed the host. But Kubernetes itself runs on the same VM here, making the cluster a tenant of the system it is supposed to manage -- which is why kubelet's ownMemoryPressure` condition appeared in the logs as a symptom of the problem it was meant to prevent.

The debugging :

Where the build environment, runner configuration, and service state were in constant flux.

The same class of errors – command not found, Nix daemon socket failures, Cachix auth failures, OOM kills, and stuck jobs – recurred across multiple attempts not because the fixes were wrong, but because each fix exposed the next layer of drift between the declared configuration and the running system.

• Reasoning the recurring errors

  • The command not found errors for grep, which, and ls inside the CI container were fixed at least twice during this process. They recurred because the fix – mounting /nix:/nix:ro as a single volume – overwrote the container’s own Nix store, destroying its internal binaries. Reverting to individual mounts (/nix/store:/nix/store:ro, /nix/var/nix/db:/nix/var/nix/db:ro) restored the container’s tools, but the daemon socket mount also needed :ro removed to allow IPC. Each of these changes required a nixos-rebuild switch to propagate from gitlab-runner.nix into the running config.toml, and a service restart to apply. The window between declaration and activation was where drift lived.
  • Stuck CI jobs were caused by the runner losing contact with GitLab after an OOM event restarted the gitlab-runner service with a new system ID. GitLab continued showing the old runner as online while the new instance was pending Protected branch approval. The fix was to assign and mark the new runner instance as Protected in the GitLab UI – a manual step that cannot be automated without a GitLab API token, and that must be repeated after every runner re-registration.

Speed of deployment

Cachix

Managed CDN-backed binary cache. Fast substitution via global edge network. Used as the primary pull cache for both pipelines.

before_script:
  - cachix authtoken "$CACHIX_AUTH_TOKEN"
  - cachix use "$CACHIX_CACHE"

Push after build host CI uses nix path-info --recursive, blog CI uses nix develop --print-out-paths:

# Host CI  full closure push
nix path-info --recursive "$spec" \
  | xargs cachix push -j 2 -n 2 $CACHIX_CACHE_NAME

# Blog CI  devShell closure push
nix develop .#ci --print-out-paths 2>/dev/null \
  | xargs nix-store -qR \
  | cachix push -j 2 -n 2 "$CACHIX_CACHE"

Atticd

Self-hosted binary cache server backed by Cloudflare R2 (zero egress). Runs on localhost:8083 on thinkpad for host CI (shell executor), on node1 for blog CI (Docker executor with network_mode = host).

before_script:
  - attic login local http://127.0.0.1:8083 $ATTIC_TOKEN   # host CI
  - attic login local http://localhost:8083 "$ATTIC_TOKEN"  # blog CI
  - attic use local:blog-cache                               # configure as substituter

Push after build:

# Host CI
nix path-info --recursive "$spec" \
  | xargs attic push -j 2 dev-cache || true

# Blog CI
nix develop .#ci --print-out-paths 2>/dev/null \
  | xargs nix-store -qR \
  | xargs attic push -j 2 blog-cache || true

R2 Storage

Single bucket attic-cache serves both logical caches. No Cloudflare UI changes needed when adding new caches Atticd routes by key prefix internally.

# Create logical cache  one time setup
attic cache create blog-cache
attic cache create dev-cache

# Verify
attic cache info blog-cache
attic cache info dev-cache

Auto-Cancel Redundant Pipelines

workflow:
  auto_cancel:
    on_new_commit: interruptible

pages:
  interruptible: true

Cancels the older pipeline immediately when a newer commit is pushed. Cachix push is atomic per-path a cancelled mid-push does not corrupt the cache.

See Architecture

The AWS Credential Chain Problem

Attic uses the official aws-sdk-s3 Rust crate (1.96.0) with aws-config (1.8.1). The storage backend initialises via:

let shared_config = aws_config::load_defaults(BehaviorVersion::v2025_01_17()).await;

This runs the full AWS credential provider chain — including an EC2 IMDS lookup at IP. On a laptop or any non-EC2 host, this times out after 5 seconds before falling back to environment variables.

The credential config in s3.rs shows why env vars are the right approach for R2:

if let Some(credentials) = &config.credentials {
    builder = builder.credentials_provider(Credentials::new(
        &credentials.access_key_id,
        &credentials.secret_access_key,
        None,
        None,
        "s3",
    ));
}

If credentials is not set in atticd.toml, the SDK falls through to env vars. Either works — but env vars via agenix keeps secrets out of the nix store. —

Execution Context and Network Identity

This was the least obvious class of failure, and the one that explained the most.

Atticd binds to 0.0.0.0:8083 on node1. The address 0.0.0.0 is a bind directive, not a routable address. What is actually reachable depends on where the request originates:

• From the host machine: 127.0.0.1:8083 resolves to node1 via Tailscale or SSH tunnel.
• From node1 itself: 127.0.0.1:8083 is the loopback interface, reaches atticd directly.
• From a Docker container on node1: 127.0.0.1 resolves to the container’s own loopback. Atticd is not there. The correct address is 172.17.0.1, the Docker bridge gateway.

The consequence was that dev-cache and blog-cache, though served by the same atticd instance, behaved as if they were different services:

The caches are logically identical. The difference is entirely in which endpoint was registered at creation time, which determined where the binary cache endpoint was advertised in nix-cache-info.

401 Is Not a Failure

A 401 Unauthorized from the cache endpoint is correct behavior for a private cache. Nix substitution (downloads) does not require authentication. attic push and API operations do. A 401 from curl confirms the server is reachable and auth is functioning. It is not a signal that the cache is broken.

The failure modes that actually indicate a problem are connection refused (wrong address or atticd not running), 404 (cache does not exist), and silent substituter fallback (Nix tried the cache, got no response, fell back to Cachix without logging why).

Root vs User Token Split

nixos-rebuild switch runs as root. The attic token lives at ~/.config/attic/config.toml for the invoking user. Root has no token. This means atticd push from a nixos-rebuild context silently fails – the build succeeds, the cache is not populated, and the next CI run pulls from Cachix instead.

The fix is either to run attic login as root separately, or to push explicitly from the CI job as the user who has the token. The latter is preferable: it keeps the push path consistent and does not require managing root credentials.

The Three Gaps

Looking across all the failures, they fall into three categories:

Declared vs operational state. Nix captures what should exist. It cannot capture what is currently running, what credentials are live, or what the OOM killer removed. journalctl, systemctl, and dmesg are the tools that close this gap, and they have to be used reactively.

Network identity vs network address. 127.0.0.1 means different things in different execution contexts. Configuration written for one context silently fails in another. The fix is to treat network endpoints as context-dependent and make the execution environment explicit in CI configuration rather than assuming a shared namespace.

Expected errors vs actual failures. 401 from atticd, a dropped Tailscale connection during nix copy, and a 429 from an RSS feed all look like failures. Only one of them is. Distinguishing expected behavior from actual failure modes requires knowing what the system is supposed to do, which is not always documented in error messages.

See Architecture

Copyright (C) 2013-2026 Xameer . All rights reserved.