Skip to content

Commit c1a0dd1

Browse files
thphamthpham
authored
✨ feat(deploy): add bare metal deployment support with NixOS live ISO workflow (#10)
Add comprehensive bare metal deployment support for systems where kexec fails (common on AMD hardware). Key changes: deploy.sh: - Add --user flag for non-root SSH users (nixos for live ISO) - Add --env-password flag for SSHPASS-based authentication - Add --no-kexec flag to skip kexec phase (--phases disko,install,reboot) base.nix: - Switch to systemd-networkd for consistent networking with microvm bridges - Disable wait-online service to prevent boot blocking - Add SSH authorized keys configuration for fleet access flake.nix: - Add sshpass to devShell for password-based SSH README.md: - Document complete bare metal deployment workflow - Add sops-nix secrets management instructions (ssh-to-age, updatekeys) - Document provider types, command reference, troubleshooting
1 parent 6e9560d commit c1a0dd1

7 files changed

Lines changed: 434 additions & 28 deletions

File tree

‎deploy/README.md‎

Lines changed: 306 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,306 @@
1+
# Deployment Guide
2+
3+
This guide covers deploying NixOS with fireactions to various machine types using `nixos-anywhere` for initial installation and `colmena` for ongoing fleet management.
4+
5+
## Prerequisites
6+
7+
1. **Management machine** with Nix and flakes enabled
8+
2. **Target machine** accessible via SSH (rescue mode, live ISO, or existing Linux)
9+
3. **SSH public key** added to `deploy/base.nix` (see [SSH Key Setup](#ssh-key-setup))
10+
4. **sops-nix** configured with your admin age key (see `secrets/.sops.yaml`)
11+
12+
## SSH Key Setup
13+
14+
Before deploying, add your SSH public key to `deploy/base.nix`:
15+
16+
```bash
17+
# Get your public key
18+
cat ~/.ssh/id_ed25519.pub
19+
```
20+
21+
Edit `deploy/base.nix` line ~45:
22+
23+
```nix
24+
users.users.root.openssh.authorizedKeys.keys = [
25+
"ssh-ed25519 AAAAC3Nza... your-key-here"
26+
];
27+
```
28+
29+
## Deployment Scenarios
30+
31+
### Cloud VMs (DigitalOcean, Hetzner)
32+
33+
Cloud VMs typically have SSH access pre-configured in rescue mode.
34+
35+
```bash
36+
# DigitalOcean
37+
./deploy/deploy.sh --provider do --name do-runner-1 \
38+
--tags dev,github-runners,fireactions-small \
39+
167.71.100.50
40+
41+
# Hetzner
42+
./deploy/deploy.sh --provider hetzner --name hetzner-runner-1 \
43+
--tags prod,github-runners,fireactions-medium \
44+
95.217.xxx.xxx
45+
```
46+
47+
### Bare Metal / Desktop from NixOS Live ISO
48+
49+
For bare-metal machines or desktops where kexec may fail (common on AMD hardware), boot from a NixOS live ISO first.
50+
51+
#### Step 1: Prepare Bootable USB
52+
53+
Download NixOS minimal ISO from https://nixos.org/download and write to USB:
54+
55+
```bash
56+
# macOS
57+
sudo dd if=nixos-minimal-*.iso of=/dev/diskX bs=4M
58+
59+
# Linux
60+
sudo dd if=nixos-minimal-*.iso of=/dev/sdX bs=4M status=progress
61+
```
62+
63+
#### Step 2: Boot Target and Enable SSH
64+
65+
On the target machine (physical access required):
66+
67+
```bash
68+
# Set password for nixos user
69+
passwd
70+
71+
# Get IP address
72+
ip addr
73+
```
74+
75+
#### Step 3: Deploy from Management Machine
76+
77+
```bash
78+
# Enter dev shell (includes sshpass)
79+
nix develop
80+
81+
# Deploy with password authentication
82+
SSHPASS='your-password' ./deploy/deploy.sh \
83+
--provider nvme \
84+
--name nixtower \
85+
--user nixos \
86+
--env-password \
87+
--tags dev,github-runners,fireactions-small \
88+
192.168.55.56
89+
```
90+
91+
The `--user nixos` flag is required because the NixOS live ISO uses `nixos` as the default user, not `root`.
92+
93+
nixos-anywhere automatically detects the NixOS installer environment and skips kexec (which can fail on AMD hardware).
94+
95+
#### Step 4: Wait for Reboot
96+
97+
After successful deployment:
98+
99+
- Machine reboots automatically
100+
- IP may change (check your DHCP server)
101+
- Update `hosts/registry.json` if IP changed
102+
103+
#### Step 5: Register Host for Secrets (sops-nix)
104+
105+
For the new host to decrypt secrets (runner tokens, API keys, etc.), you must add its age key to `secrets/.sops.yaml`:
106+
107+
```bash
108+
# Get the host's age key from its SSH host key
109+
ssh-keyscan <host-ip> 2>/dev/null | ssh-to-age
110+
# Or if you can SSH directly:
111+
ssh root@<host-ip> "cat /etc/ssh/ssh_host_ed25519_key.pub" | ssh-to-age
112+
```
113+
114+
Edit `secrets/.sops.yaml`:
115+
116+
```yaml
117+
keys:
118+
# ... existing keys ...
119+
- &my-new-host age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
120+
121+
creation_rules:
122+
- path_regex: secrets\.yaml$
123+
key_groups:
124+
- age:
125+
- *admin
126+
- *my-new-host # Add the new host anchor here
127+
```
128+
129+
Re-encrypt secrets so the new host can decrypt them:
130+
131+
```bash
132+
# Enter dev shell (includes sops)
133+
nix develop
134+
135+
# Re-encrypt with the new key
136+
sops updatekeys secrets/secrets.yaml
137+
```
138+
139+
Deploy to push the updated configuration:
140+
141+
```bash
142+
colmena apply --on my-new-host --build-on-target
143+
```
144+
145+
### Existing NixOS System (with working kexec)
146+
147+
If kexec works on your hardware:
148+
149+
```bash
150+
./deploy/deploy.sh --provider nvme --name my-server 192.168.1.100
151+
```
152+
153+
If kexec fails (machine freezes), use the [Live ISO method](#bare-metal--desktop-from-nixos-live-iso) instead.
154+
155+
## Provider Types
156+
157+
| Provider | Disk Device | Use Case |
158+
| ----------- | -------------- | -------------------------------------------- |
159+
| `do` | `/dev/vda` | DigitalOcean droplets |
160+
| `hetzner` | `/dev/sda` | Hetzner Cloud/Dedicated |
161+
| `generic` | `/dev/vda` | Generic KVM/QEMU VMs |
162+
| `nvme` | `/dev/nvme0n1` | NVMe-based systems (modern desktops/servers) |
163+
| `baremetal` | `/dev/sda` | Bare metal with SATA/SAS |
164+
| `auto` | auto-detect | Generates hardware-configuration.nix |
165+
166+
## Command Reference
167+
168+
### deploy.sh Options
169+
170+
```
171+
Usage: ./deploy.sh [OPTIONS] <ip>
172+
./deploy.sh list
173+
./deploy.sh unregister <name>
174+
175+
Options:
176+
--provider, -p TYPE Provider: do, hetzner, generic, nvme, baremetal (default), auto
177+
--name, -n NAME Host name (default: auto-generated)
178+
--tags, -t TAG1,TAG2 Comma-separated tags for profile selection
179+
--arch, -a ARCH Architecture: x86_64 (default) or aarch64
180+
--user, -u USER SSH user (default: root, use 'nixos' for live ISO)
181+
--env-password Use password from SSHPASS environment variable
182+
--no-kexec Skip kexec phase (manually, usually auto-detected)
183+
```
184+
185+
### List Registered Hosts
186+
187+
```bash
188+
./deploy/deploy.sh list
189+
```
190+
191+
### Remove Host from Registry
192+
193+
```bash
194+
./deploy/deploy.sh unregister my-host
195+
```
196+
197+
## Fleet Management with Colmena
198+
199+
After initial deployment, use colmena for configuration updates:
200+
201+
```bash
202+
# Update single host
203+
colmena apply --on nixtower --build-on-target
204+
205+
# Update all hosts with a specific tag
206+
colmena apply --on @dev --build-on-target
207+
208+
# Update all hosts
209+
colmena apply --build-on-target
210+
```
211+
212+
## Tag-Based Profiles
213+
214+
Profiles are applied based on tags specified during deployment:
215+
216+
### Environment Profiles
217+
218+
- `dev` - Debug mode, verbose logging
219+
- `prod` - Production hardening
220+
221+
### Workload Profiles
222+
223+
- `github-runners` - GitHub Actions runners
224+
- `gitea-runners` - Gitea Actions runners
225+
- `gitlab-runners` - GitLab CI runners
226+
227+
### Size Profiles
228+
229+
- `fireactions-small` - 1GB RAM, 1 vCPU, 1-2 runners
230+
- `fireactions-medium` - 2GB RAM, 2 vCPU, 2-5 runners
231+
- `fireactions-large` - 4GB RAM, 4 vCPU, 5-10 runners
232+
- (similarly for `fireteact-*` and `fireglab-*`)
233+
234+
### Example Combinations
235+
236+
```bash
237+
# GitHub Actions only (small)
238+
--tags dev,github-runners,fireactions-small
239+
240+
# Multi-platform runners
241+
--tags prod,github-runners,gitlab-runners,fireactions-medium,fireglab-medium
242+
243+
# With registry cache
244+
--tags prod,github-runners,fireactions-large,registry-cache
245+
```
246+
247+
## Troubleshooting
248+
249+
### SSH Connection Failed
250+
251+
```
252+
ERROR: Cannot connect to nixos@192.168.55.64 with password
253+
```
254+
255+
**Solutions:**
256+
257+
1. Verify password is set on target: `passwd`
258+
2. Check IP address: `ip addr`
259+
3. Ensure SSH is running: `systemctl status sshd`
260+
261+
### Kexec Fails / Machine Freezes
262+
263+
Some hardware (especially AMD) doesn't support kexec well. Use the NixOS live ISO method instead - nixos-anywhere automatically skips kexec when it detects an installer environment.
264+
265+
### Permission Denied After Deployment
266+
267+
```
268+
root@192.168.55.56: Permission denied (publickey)
269+
```
270+
271+
Your SSH key is not in the deployed system. Add it to `deploy/base.nix` and redeploy, or manually add it on the target:
272+
273+
```bash
274+
# On target machine (physical access)
275+
mkdir -p /root/.ssh
276+
echo "ssh-ed25519 AAAAC3Nza... your-key" >> /root/.ssh/authorized_keys
277+
chmod 700 /root/.ssh
278+
chmod 600 /root/.ssh/authorized_keys
279+
```
280+
281+
### IP Changed After Reboot
282+
283+
DHCP may assign a different IP. Update `hosts/registry.json`:
284+
285+
```json
286+
{
287+
"nixtower": {
288+
"hostname": "192.168.55.56", // <- Update this
289+
...
290+
}
291+
}
292+
```
293+
294+
## File Overview
295+
296+
| File | Purpose |
297+
| ------------------------- | --------------------------------------------- |
298+
| `deploy.sh` | Main deployment script |
299+
| `base.nix` | Shared config (boot, SSH, firewall, packages) |
300+
| `disko.nix` | Disk partitioning (LVM on GPT) |
301+
| `secrets.nix` | sops-nix configuration |
302+
| `digitalocean.nix` | DigitalOcean-specific settings |
303+
| `configuration.nix` | Legacy entry point |
304+
| `../secrets/.sops.yaml` | Age keys for secret encryption |
305+
| `../secrets/secrets.yaml` | Encrypted secrets (runner tokens, API keys) |
306+
| `../hosts/registry.json` | Deployed hosts registry |

‎deploy/base.nix‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,14 +23,34 @@
2323

2424
# Networking (hostName set by hosts/default.nix safety module)
2525
networking = {
26-
useDHCP = lib.mkDefault true;
26+
# Use systemd-networkd for consistent networking with microvm bridges
27+
useNetworkd = lib.mkDefault true;
28+
useDHCP = lib.mkDefault false; # Handled by networkd below
29+
2730
firewall = {
2831
enable = true;
2932
allowedTCPPorts = [ 22 ];
3033
logRefusedConnections = false; # Reduce noise from port scans
3134
};
3235
};
3336

37+
# Configure systemd-networkd for physical ethernet interfaces
38+
# This works alongside microvm-base bridge configuration
39+
systemd.network = {
40+
enable = lib.mkDefault true;
41+
wait-online.enable = false; # Don't block boot waiting for network
42+
43+
# DHCP on physical ethernet interfaces
44+
# Matches common naming: enp*, eno*, ens*, eth* (but not bridges like fireactions0)
45+
networks."20-ethernet" = {
46+
matchConfig.Name = "en* eth*";
47+
networkConfig = {
48+
DHCP = "yes";
49+
IPv6AcceptRA = true;
50+
};
51+
};
52+
};
53+
3454
# SSH configuration
3555
services.openssh = {
3656
enable = true;
@@ -40,6 +60,14 @@
4060
};
4161
};
4262

63+
# Management SSH keys for fleet access
64+
# Add your public key(s) here to enable colmena deployments
65+
users.users.root.openssh.authorizedKeys.keys = [
66+
# TODO: Add your SSH public key here
67+
# "ssh-ed25519 AAAAC3Nza... user@host"
68+
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOPK49ZsqsVVeKdpFuT4aJn4oNYsyTPvSM7Insw2wR2k"
69+
];
70+
4371
# Fail2ban for SSH brute-force protection
4472
services.fail2ban = {
4573
enable = true;

0 commit comments

Comments
 (0)