CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Goal

krg-infra is the KastnerRG infrastructure monorepo, replacing the old Ansible infrastructure at fabricant-prod and waiter. Layers are split by configuration tool (not by guest/host role — some NixOS machines are physical):

nix/ — every machine configured by NixOS (the flake): physical hosts (waiter) and Proxmox VMs (krg-prod, e4e-prod, krg-ldap). Machines are composed from profile modules — no per-host playbooks.
ansible/ — the Proxmox/Debian hypervisor hosts those VMs run on.
terraform/ — the OpenTofu layer: things driven through a Terraform/ Web-API provider rather than NixOS or Ansible. One root module (own state + creds) per target — not one combined config (separate blast radius). Targets: e4e-nas/ (Synology DSM), authentik/ (SSO config), openbao/ (secrets-engine structure + AppRole auth), grafana/ (dashboards/datasources/SSO) — all built out (real .tf, not stubs). authentik/ + openbao/ + grafana/ manage the config of services deployed elsewhere (Authentik & Grafana = compose on krg-prod; OpenBao = krg-vault), not their deployment. e4e-nas is a hybrid: the synology provider only covers Container Manager / packages / tasks / files / VMs; identity (AD join), shares/ACLs, firewall, SSH, snapshots and DSM updates have no API and live in the runbook docs/e4e-nas-dsm.md (DSM is a proprietary appliance — SSH-level edits don't survive DSM updates, so prefer UI settings).

This whole rebuild is incident-response driven: a Proxmox host's root SSH was dictionary-attacked. The hypervisors had no config management — ansible/ closes that gap (secure + up to date by default), and the old AD (inside the blast radius) is being rebuilt clean as a new Samba AD forest on krg-ldap.

Target feature set (from the old Ansible repos):

fabricant-prod: production services — Traefik, Authentik (SSO), Grafana/Prometheus/Loki, Blackbox, PostgreSQL, Outline, MLflow, Label Studio, node/IPMI exporters, firewall, unattended upgrades. (These are the lab-wide tools now on krg-prod; E4E project-specific services go on e4e-prod.)
waiter: research/compute at 137.110.161.67 — NVIDIA CUDA + Container Toolkit, FPGA tooling (Vivado, Vitis, Verilator; opt-in), XRDP+XFCE desktop (opt-in, gated on FPGA), CrowdSec, native node/IPMI exporters + DCGM exporter (Docker) — blackbox now lives on krg-prod — and ZFS-on-root with ZFS auto-snapshots (replacing the old btrfs/snapper). The legacy Ubuntu setup used btrfs + Docker-based node/blackbox; the rebuild moved those to the patterns above.

Operating principle: IaC-strict

Every live change against managed infrastructure (NAS, NixOS hosts, Proxmox, etc.) flows through IaC. When something needs to change on a managed machine, the fix is:

Extend the spec (spec/e4e-nas/*.yml, nix/..., terraform/...).
Extend the role / module / resource that consumes it (apply_*.py subcommand, NixOS module, terraform resource).
Commit + push.
Apply via the appropriate orchestrator (ansible-playbook, nixos-rebuild, tofu apply).

Do not propose live mutations as the fix itself:

❌ "Click here in DSM web → Apply"
❌ "Run synowebapi --exec ... method=set ... on the NAS to fix it"
❌ "Edit /etc/... directly via SSH"
❌ "Manually create the bucket / share / cert / firewall rule"

Even when the goal is to "just unblock today," the right framing is "extend the role NOW, it's the same 5 minutes either way." UI-driven or shell-driven fixes are invisible to the spec → they're drift the moment they land (ADR 0001: git is truth, UI = drift).

Discovery is the only exception. Read-only probes, Chrome DevTools network captures of a wizard click to learn the API shape, synowebapi --exec ... method=get, log greps, etc. are fine — flag them as discovery, then port the findings into the role / spec / module before applying. Capturing a wizard's POST shape is research; clicking the wizard's "Save" as the fix delivery is not.

This applies broadly to nix/, ansible/, terraform/ work — not just the NAS. The Synology surface had the most run-ins with it because DSM has the richest "click here to fix it" affordance, but the rule is global.

Common Commands

The flake lives in nix/. Run from the repo root with the ./nix ref shown below, or cd nix and drop the prefix.

# Validate the flake (Nix syntax + module type checking)
nix flake check ./nix

# Build a system config without deploying
nix build ./nix#nixosConfigurations.krg-prod.config.system.build.toplevel
nix build ./nix#nixosConfigurations.waiter.config.system.build.toplevel

# Inspect a config value
nix eval ./nix#nixosConfigurations.krg-prod.config.networking.hostName

# Deploy to the current machine
sudo nixos-rebuild switch --flake ./nix#krg-prod

# Deploy remotely (new nixos-rebuild: --sudo, not --use-remote-sudo)
nixos-rebuild switch --flake ./nix#krg-prod --target-host krg-admin@krg-prod.ucsd.edu --sudo --ask-sudo-password
nixos-rebuild switch --flake ./nix#waiter   --target-host krg-admin@137.110.161.67 --sudo --ask-sudo-password

# Update flake inputs (run inside nix/)
cd nix && nix flake update          # or: nix flake update nixpkgs

# Format .nix files (alejandra, via treefmt). This is the SAME formatter the
# `nix flake check` gate enforces (checks.formatting), so always format before
# committing — unformatted Nix fails CI.
cd nix && nix fmt

# --- Proxmox hosts (ansible/) ---
cd ansible
ansible-galaxy collection install -r requirements.yml
ansible-playbook playbooks/site.yml --check     # dry run
ansible-playbook playbooks/site.yml

Repository Structure

krg-infra/
  CLAUDE.md  README.md  LICENSE  .github/workflows/build.yml
  nix/                             # NixOS machines (physical + Proxmox guests)
    flake.nix                      # inputs + nixosConfigurations (krg-prod, e4e-prod, waiter, kastner-ml, krg-ldap, krg-vault, krg-deploy, krg-nat)
    keys/admins.json               # SHARED admin SSH keys — read by nix + ansible
    networks/trusted.json          # SHARED trusted nets / Proxmox IPSets — read by nix + ansible + PVE cluster.fw
    modules/
      docker.nix  users.nix  nix-ld.nix  impermanence.nix
      zfs.nix                      # ZFS support + auto-scrub + auto-snapshot retention (krg.zfs); WHICH datasets/cadence is per-dataset via com.sun:auto-snapshot props. NixOS hosts run the timers; fabricant (Proxmox) runs the same tool via ansible nfs_server
      nfs-home.nix                 # mount /home from NFS (krg.nfsHome) — AD user homes off the local ZFS root (enables impermanence)
      scratch.nix  scratch/        # /scratch/<lab> on PLAIN ZFS (krg.scratch) — scratchpool (striped HDD + NVMe special/L2ARC); daily cold-file overflow to NFS + self-service scratch-restore (modules/scratch/*.py). Replaced autotier (FUSE; crashed under concurrent training reads)
      local-cache.nix              # node-local /local/<user> (krg.localCache) — IDE servers + cache class OFF the NFS /home (durable NVMe dataset, no FUSE/NFS)
      samba-ad.nix                 # Samba AD domain controller (samba4Full daemon, krb5.conf, DNS/resolver, AD ports)
      sssd-ad-client.nix           # SSSD AD client (krg.adClient) — every host joins KRG.LOCAL; key-only SSH with keys served from AD (sss_ssh_authorizedkeys + OpenSSH-LPK)
      ad-group-sync.nix            # generic AD-group → local-group bridge (krg.adGroupSync) — re-derives a fixed-GID local group's members from one+ AD groups via a oneshot+10-min timer. Consumed by nvidia.nix (cudaAccessGroups → GPU device group) and docker.nix (accessGroups → docker daemon group)
      security/{firewall,crowdsec,crowdsec-bouncer,oec-qualys-trellix}.nix   # firewall is the single switch; crowdsec is the ban layer
      services/{compose-stack,node-exporter,ipmi-exporter}.nix
      hardware/{nvidia,fpga}.nix   desktop/xrdp.nix
    profiles/
      base.nix                     # every host: SSH hardening, auto-upgrade, OEC + CrowdSec + node-exporter + in-guest firewall; isVM enables qemu-guest-agent
      server.nix                   # krg-prod / e4e-prod role (docker, compose, ipmi exporter)
      compute.nix                  # waiter role (physical; NVIDIA/FPGA/XRDP/ZFS)
      directory.nix                # krg-ldap role: Samba AD DC (realm KRG.LOCAL)
    hosts/{krg-prod,e4e-prod,waiter,krg-ldap}/{default,hardware-configuration}.nix
    users/admin.nix                # local break-glass admin (krg-admin/e4e-admin); home /var/lib/<account> (OFF /home, so an NFS /home mount can't shadow it); keys from keys/admins.json; human users come from Samba AD
    docker-compose/                # compose stacks mounted by the flake
      krg-prod/                    # the lab-wide services stack (Traefik, Authentik, Grafana, …) wired into nix/hosts/krg-prod/default.nix
      dcgm-exporter/               # standalone NVIDIA DCGM exporter, wired by modules/hardware/nvidia.nix (krg.nvidia.dcgmExporter) — there is no per-host waiter/ dir, just this
  ansible/                         # Proxmox hypervisor hosts (Debian/PVE)
    ansible.cfg  requirements.yml
    inventory/
      hosts.yml                    # the Proxmox hosts (group: proxmox) — currently one host, "fabricant"
      group_vars/{all,proxmox}.yml # next to the inventory (so ansible-playbook loads it): all.yml = generic baseline (keys/trusted nets via the shared files); proxmox.yml = PVE-specific
      host_vars/fabricant.yml      # fabricant-ONLY vars (NFS shares, ZFS limits, host.fw rules)
    playbooks/site.yml             # all hosts → base; proxmox group → proxmox_firewall; fabricant → zfs_limits + nfs_server
    roles/
      base/                        # THE baseline: OS basics (timezone, packages incl tmux, unattended upgrades, sysctl) + composes the security/monitoring roles below (import_role, ordered: krg_admin → ssh_hardening → crowdsec → monitoring → oec)
      krg_admin/                   # key-only sudo krg-admin (mirrors nix/users/admin.nix)
      ssh_hardening/               # disable password auth, root key-only (the breach fix)
      crowdsec/                    # community CTI + local scenarios — the fleet brute-force/ban layer
      monitoring/                  # node + ipmi exporters (systemd) — on every host via base
      oec_qualys_trellix/          # campus-mandated Qualys + Trellix (set oec_installer) — via base
      proxmox_firewall/            # PVE cluster.fw + per-guest <vmid>.fw + per-node host.fw (host rules eval before cluster rules; proxmox group, separate play)
      zfs_limits/                  # quota/reservation on EXISTING ZFS datasets — caps VM storage so user data wins (fabricant ONLY play)
      nfs_server/                  # NFSv4 exports on ZFS datasets under <pool>/nfs (fabricant ONLY play; NFS tcp/2049 opened via fabricant host.fw); ALSO schedules zfs-auto-snapshot (systemd timers) for the opted-in shares — Proxmox has no NixOS services.zfs.autoSnapshot, so retention here MIRRORS nix krg.zfs.autoSnapshot, scoped opt-in via --default-exclude
  terraform/                       # OpenTofu layer — one root module (own state+creds) per target
    README.md  .gitignore          # layer conventions; .gitignore is top-level (covers all targets: *.tfstate*, *.tfvars)
    e4e-nas/                       # Synology DSM (synology-community/synology): containers/packages/scheduler.tf + FileStation folders; Garage CONFIG lives in the ansible role (ADR 0007), here = just the dirs/folders it needs
    authentik/                     # BUILT (goauthentik/authentik): SSO apps/providers/flows/groups/LDAP-outpost (applications_{krg,e4e}.tf, data.tf, groups.tf, ldap.tf, outpost.tf) + writes generated OIDC client secrets to OpenBao (vault_secrets.tf, roster_secrets.tf)
    openbao/                       # BUILT (was "vault/"; openbao provider): KV-v2 mount + AppRole auth + per-role policies (krg-deploy, krg-prod) — structure not values (main.tf)
    grafana/                       # BUILT (grafana/grafana): dashboards/datasources/folders/teams/SSO — reads grafana-oidc from OpenBao (sso.tf)
    temporal/                      # BUILT (temporal provider): namespaces/config on the lab-wide Temporal cluster (temporalio/auto-setup on krg-prod, workflows.krg.ucsd.edu) — declared here, not via one-off tctl
    secrets/                       # BUILT (openbao provider): mints the OIDC client-secrets + DB passwords krg-prod services need at startup into OpenBao (secret/krg-prod/authentik-managed/*) — runs early (before P2 vault-agent renders consume them); fixes the deploy-DAG ordering
  spec/e4e-nas/                     # declarative DSM source of truth (shares/groups/smb-globals/nfs-exports/acls/users/garage) — consumed by ansible synology_* roles; seeded from docs/e4e-nas-dsm.md
  docs/
    creating-a-user.md             # AD user-creation runbook
    e4e-nas-dsm.md                 # DSM runbook: the NAS settings with no API (AD join, shares/ACLs, firewall, SSH, snapshots, updates) — → break-glass once spec/ seeded
    krg-prod-iac.md                # krg-prod IaC architecture + NAS standup plan (how the spec maps onto this repo)
    adr/                           # Architecture Decision Records — 0001 IaC-source-of-truth, 0002 Garage, 0003 Garage-on-NAS, 0004 VM-IO-budget, 0005 repo-integration/OpenTofu/krg-deploy, 0006 no-OEC-on-DSM, 0007 DSM-ansible-not-terraform, 0008 e4e-prod-tenant-platform, 0009 lab-internal-PKI-AD, 0010 AD-structure-as-IaC, 0011 cross-layer-deploy-ordering, 0012 endpoint-device-management, 0013 SSO-Authentik-front-door, 0014 proxmox-auth-ldap-outpost, 0015 fleet-wide-container-logs-Loki (Proposed), 0016 developed-apps-one-trust-tier, 0017 incus-nat-self-serve-platform (re-realizes 0008), 0018 monitoring-long-running-vms, 0019 proxmox-to-incus-migration, 0020 tenant-deploy-contract-mkTenant, 0021 tenant-TLS-vault-agent-and-secret-zero, 0022 tenant-runners-github-app-broker (Proposed), 0023 tenant-temporal-client-cert (Proposed)

Naming note: fabricant now refers only to the Proxmox host (hypervisor, in the ansible inventory). The old nix fabricant services config was split into krg-prod (lab-wide tools) and e4e-prod (E4E project services).

Architecture: Key Patterns

0. Firewall ownership (defense-in-depth, split by layer)

Each layer owns the firewall concern it's best at, so they don't drift:

In-guest NixOS firewall (krg.firewall) — on EVERY host, VMs included. It owns which ports a service exposes (e.g. samba-ad.nix declares the AD port set) and gives CrowdSec a backend (the countermeasure to the dictionary attack that drove this rebuild). profiles/base.nix sets it mkDefault true.
Proxmox host firewall (ansible/) — additive perimeter. Owns which sources may reach a VM, plus containment. Services → ucsd/sealab; compute (waiter) → public SSH (protected by key-only + CrowdSec). Does not replace the in-guest layer.

0b. Shared data across layers

Values needed by both nix and ansible live in one file each, under nix/ (the flake can only read its own subtree); ansible reads them across the repo:

nix/keys/admins.json — admin SSH public keys (not secret).
nix/networks/trusted.json — trusted nets / Proxmox IPSets + monitoring host. Edit these, not the per-layer copies.

1. NixOS modules vs Docker Compose

Services that were native systemd in Ansible (node_exporter, ipmi_exporter) use native NixOS services.prometheus.exporters.* modules. Everything else stays as Docker Compose stacks managed by krg.composeStacks.

The compose-stack module runs each stack as a systemd oneshot service with docker compose --project-directory <workingDir> -f <nix-store-path> up -d. The --project-directory flag makes Docker Compose resolve relative volume paths (like ./.secrets/foo.txt) against the working directory (e.g. /var/lib/krg/krg-prod/), not the Nix store. The compose files stay read-only in the store; runtime data (databases, secrets, config) lives in the working directory.

2. Compose file `include:` and the directory reference pattern

compose.yml uses Docker Compose's include: directive to pull in sub-stacks. For this to work when the compose files are in the Nix store, the entire nix/docker-compose/krg-prod/ directory must be in the same store path. Always reference the directory, not individual files:

# In nix/hosts/krg-prod/default.nix — correct pattern
let composeDir = ../../docker-compose/krg-prod; in
{
  krg.composeStacks.krg-prod.composeFiles = [ "${composeDir}/compose.yml" ];
}
# The whole docker-compose/krg-prod/ directory is copied to the store,
# so include: can find compose.authentik.yml etc. alongside compose.yml.

3. Adding a new machine (NixOS)

Run nixos-generate-config --show-hardware-config on the target; save as nix/hosts/<name>/hardware-configuration.nix
Create nix/hosts/<name>/default.nix importing the appropriate profile plus any krg.composeStacks
Add the host to nix/flake.nix under nixosConfigurations
git add the new files (a flake only sees git-tracked files), nix flake check ./nix, then deploy with nixos-rebuild switch --flake ./nix#<name> --target-host ...

4. Authentik app icons

Every third-party application surfaced in Authentik must ship an app-tile icon. An iconless tile renders as a generic placeholder and reads as a broken/untrusted link on the user dashboard. When you add an app to terraform/authentik/applications_*.tf, add its icon in the same change: drop the SVG in nix/docker-compose/krg-prod/authentik/media-icons/ and set meta_icon = "krg-icons/<file>.svg". If no clean upstream logo exists yet, record it in that dir's README.md "Not yet iconed" list so the gap is tracked, not forgotten. See nix/docker-compose/krg-prod/authentik/media-icons/README.md for the wiring.

Secrets (Pre-Vault)

Secrets are not managed by Nix yet (future: OpenBao/HashiCorp Vault on krg-vault). Before starting each compose stack, manually create the required files in the working directory. Each host's default.nix lists required secrets in a comment. (Vaultwarden is still in use — the earlier "drop Bitwarden" decision was reversed; it serves vaultwarden.krg.ucsd.edu.)

krg-prod secrets in /var/lib/krg/krg-prod/.secrets/:

authentik_postgres_admin_password.txt
authentik_admin_password.env (AUTHENTIK_SECRET_KEY=... and AUTHENTIK_POSTGRESQL__PASSWORD=...)
authentik_traefik_token.env
gf_admin_password.txt
outline_secrets.env
mlflow.env (POSTGRES_PASSWORD, OIDC_* variables)

The .secrets/ directories are in .gitignore.

Runtime Config Directories

The grafana/prometheus/loki compose services mount config from the working directory. Before starting the krg-prod stack, populate:

/var/lib/krg/krg-prod/grafana/ — Grafana config
/var/lib/krg/krg-prod/loki/loki-config.yaml — Loki config
/var/lib/krg/krg-prod/loki/config.alloy — Grafana Alloy log-shipper config (replaced Promtail, EOL 2026-03-02; tails /var/log + the systemd journal)
/var/lib/krg/krg-prod/prometheus/prometheus.yml — Prometheus scrape config (targets the renamed hosts; the dead ansible_deploy_monitor :9000 job and the not-yet-provisioned kastner-ml targets are the remaining cleanups)
/var/lib/krg/krg-prod/blackbox-exporter/blackbox.yml — copy from nix/docker-compose/krg-prod/blackbox-exporter/blackbox.yml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CLAUDE.md

Project Goal

Operating principle: IaC-strict

Common Commands

Repository Structure

Architecture: Key Patterns

0. Firewall ownership (defense-in-depth, split by layer)

0b. Shared data across layers

1. NixOS modules vs Docker Compose

2. Compose file `include:` and the directory reference pattern

3. Adding a new machine (NixOS)

4. Authentik app icons

Secrets (Pre-Vault)

Runtime Config Directories

Pending Items

Uh oh!

FilesExpand file tree

CLAUDE.md

Latest commit

History

CLAUDE.md

File metadata and controls

CLAUDE.md

Project Goal

Operating principle: IaC-strict

Common Commands

Repository Structure

Architecture: Key Patterns

0. Firewall ownership (defense-in-depth, split by layer)

0b. Shared data across layers

1. NixOS modules vs Docker Compose

2. Compose file include: and the directory reference pattern

3. Adding a new machine (NixOS)

4. Authentik app icons

Secrets (Pre-Vault)

Runtime Config Directories

Pending Items

2. Compose file `include:` and the directory reference pattern