Skip to content

Commit 6d665fc

Browse files
winlinuxmattwinlinuxmatt
authored
Merge pull request #31 from winlinuxmatt/feature/longhorn-talos-setup
feat: Add Longhorn Talos setup with kubelet extraMounts
2 parents 013029e + be0545d commit 6d665fc

3 files changed

Lines changed: 243 additions & 25 deletions

File tree

‎LONGHORN_TALOS_SETUP.md‎

Lines changed: 110 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,138 @@
11
# Longhorn on Talos Linux - Setup Guide
22

3-
## Environment Check Results
3+
## Prerequisites
4+
5+
### Talos Worker Node Configuration
6+
7+
**IMPORTANT**: Worker nodes must have kubelet extraMounts configured for Longhorn to work properly. This is already configured in `cluster.tf` for all worker nodes:
8+
9+
```hcl
10+
config_patches = [
11+
yamlencode({
12+
machine = {
13+
kubelet = {
14+
extraMounts = [
15+
{
16+
destination = "/var/lib/longhorn"
17+
type = "bind"
18+
source = "/var/lib/longhorn"
19+
options = ["bind", "rshared", "rw"]
20+
}
21+
]
22+
}
23+
}
24+
})
25+
]
26+
```
427

5-
The Longhorn environment check script shows several warnings/errors that are **expected and safe to ignore** on Talos Linux:
28+
After applying Terraform changes to worker configurations, **reboot the worker nodes** for kubelet changes to take effect:
629

7-
### ✅ Safe to Ignore (Talos-specific)
30+
```bash
31+
talosctl reboot --nodes 10.0.0.73
32+
talosctl reboot --nodes 10.0.0.74
33+
talosctl reboot --nodes 10.0.0.75
34+
```
835

9-
1. **Kernel detection failures** - Talos doesn't have bash in nsenter containers
10-
2. **OS detection failures** - Same reason as above
11-
3. **iscsid service warnings** - Talos uses `ext-iscsid` instead of traditional `iscsid.service`
36+
## Quick Installation
1237

13-
### ✅ Verified Working
38+
Use the provided installation script for a complete setup:
1439

15-
All Talos worker nodes have `ext-iscsid` service running:
1640
```bash
17-
talosctl services | grep ext-iscsid
18-
# ext-iscsid Running
41+
./longhorn-install.sh
1942
```
2043

21-
## Talos Configuration Requirements
44+
This script will:
45+
1. Create the `longhorn-system` namespace with proper pod security labels
46+
2. Install Longhorn via Helm
47+
3. Expose the UI on NodePort 30080
48+
4. Configure storage disks on all worker nodes
2249

23-
Longhorn works on Talos with the following considerations:
50+
## Manual Installation
2451

25-
1. **iSCSI Support**: Talos includes `ext-iscsid` by default (verified running)
26-
2. **Kernel Version**: 6.12.57-talos (exceeds minimum requirement of 5.8)
27-
3. **Storage**: Raw block devices available on worker nodes
52+
### Step 1: Create Namespace
2853

29-
## Installation
54+
```bash
55+
kubectl create namespace longhorn-system
56+
kubectl label namespace longhorn-system \
57+
pod-security.kubernetes.io/enforce=privileged \
58+
pod-security.kubernetes.io/audit=privileged \
59+
pod-security.kubernetes.io/warn=privileged
60+
```
3061

31-
Install Longhorn using the provided values file:
62+
### Step 2: Install Longhorn
3263

3364
```bash
3465
helm repo add longhorn https://charts.longhorn.io
3566
helm repo update
3667
helm install longhorn longhorn/longhorn \
37-
--namespace longhorn-system \
38-
--create-namespace \
39-
--values longhorn-values.yaml
68+
--namespace longhorn-system \
69+
--set defaultSettings.createDefaultDiskLabeledNodes=true \
70+
--set persistence.defaultClassReplicaCount=3
71+
```
72+
73+
### Step 3: Expose UI
74+
75+
```bash
76+
kubectl patch svc longhorn-frontend -n longhorn-system \
77+
-p '{"spec": {"type": "NodePort", "ports": [{"port": 80, "targetPort": 8000, "nodePort": 30080}]}}'
78+
```
79+
80+
### Step 4: Configure Storage Disks
81+
82+
```bash
83+
# For each worker node
84+
kubectl patch nodes.longhorn.io talos-worker-01 -n longhorn-system --type='merge' \
85+
-p '{"spec":{"disks":{"default-disk":{"path":"/var/lib/longhorn","allowScheduling":true,"storageReserved":0}}}}'
86+
87+
kubectl patch nodes.longhorn.io talos-worker-02 -n longhorn-system --type='merge' \
88+
-p '{"spec":{"disks":{"default-disk":{"path":"/var/lib/longhorn","allowScheduling":true,"storageReserved":0}}}}'
89+
90+
kubectl patch nodes.longhorn.io talos-worker-03 -n longhorn-system --type='merge' \
91+
-p '{"spec":{"disks":{"default-disk":{"path":"/var/lib/longhorn","allowScheduling":true,"storageReserved":0}}}}'
4092
```
4193

42-
## Post-Installation
94+
## Access
95+
96+
- **Longhorn UI**: http://<worker-node-ip>:30080
97+
- **Worker IPs**: 10.0.0.73, 10.0.0.74, 10.0.0.75
4398

44-
1. Access Longhorn UI via NodePort on port 30080
45-
2. Manually add disks in the UI for each worker node
46-
3. Verify volume provisioning works
99+
## Verification
100+
101+
Check storage status:
102+
```bash
103+
kubectl get nodes.longhorn.io -n longhorn-system -o jsonpath='{range .items[*]}{.metadata.name}: {.status.diskStatus.default-disk.storageAvailable}{"\n"}{end}'
104+
```
105+
106+
Check all pods are running:
107+
```bash
108+
kubectl get pods -n longhorn-system
109+
```
110+
111+
## Talos-Specific Notes
112+
113+
### ✅ Safe to Ignore
114+
115+
1. **Kernel detection failures** - Talos doesn't have bash in nsenter containers
116+
2. **OS detection failures** - Same reason as above
117+
3. **iscsid service warnings** - Talos uses `ext-iscsid` instead of traditional `iscsid.service`
118+
119+
### ✅ Verified Working
120+
121+
- **iSCSI Support**: Talos includes `ext-iscsid` by default
122+
- **Kernel Version**: 6.12.57-talos (exceeds minimum requirement of 5.8)
123+
- **MountPropagation**: Enabled via kubelet extraMounts in Terraform
47124

48125
## Troubleshooting
49126

50127
If volumes fail to attach, verify:
128+
- Worker nodes were rebooted after Terraform apply
51129
- `ext-iscsid` is running: `talosctl services`
52-
- Disks are properly formatted and added in Longhorn UI
130+
- Disks are configured: `kubectl get nodes.longhorn.io -n longhorn-system`
53131
- CSI driver pods are healthy: `kubectl get pods -n longhorn-system`
132+
133+
## Uninstall
134+
135+
```bash
136+
helm uninstall longhorn -n longhorn-system
137+
kubectl delete ns longhorn-system
138+
```

‎cluster.tf‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,12 +102,29 @@ resource "talos_machine_configuration_apply" "cp_config_apply_03" {
102102

103103
# Worker Machine Configurations
104104
# All workers use the VIP as the cluster endpoint for HA
105+
# Includes Longhorn requirements: kubelet extra mounts for MountPropagation
105106
data "talos_machine_configuration" "machineconfig_worker" {
106107
cluster_name = var.cluster_name
107108
cluster_endpoint = "https://${var.cluster_vip}:6443"
108109
machine_type = "worker"
109110
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
110111
kubernetes_version = var.kubernetes_version
112+
config_patches = [
113+
yamlencode({
114+
machine = {
115+
kubelet = {
116+
extraMounts = [
117+
{
118+
destination = "/var/lib/longhorn"
119+
type = "bind"
120+
source = "/var/lib/longhorn"
121+
options = ["bind", "rshared", "rw"]
122+
}
123+
]
124+
}
125+
}
126+
})
127+
]
111128
}
112129

113130
data "talos_machine_configuration" "machineconfig_worker_02" {
@@ -116,6 +133,22 @@ data "talos_machine_configuration" "machineconfig_worker_02" {
116133
machine_type = "worker"
117134
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
118135
kubernetes_version = var.kubernetes_version
136+
config_patches = [
137+
yamlencode({
138+
machine = {
139+
kubelet = {
140+
extraMounts = [
141+
{
142+
destination = "/var/lib/longhorn"
143+
type = "bind"
144+
source = "/var/lib/longhorn"
145+
options = ["bind", "rshared", "rw"]
146+
}
147+
]
148+
}
149+
}
150+
})
151+
]
119152
}
120153

121154
data "talos_machine_configuration" "machineconfig_worker_03" {
@@ -124,6 +157,22 @@ data "talos_machine_configuration" "machineconfig_worker_03" {
124157
machine_type = "worker"
125158
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
126159
kubernetes_version = var.kubernetes_version
160+
config_patches = [
161+
yamlencode({
162+
machine = {
163+
kubelet = {
164+
extraMounts = [
165+
{
166+
destination = "/var/lib/longhorn"
167+
type = "bind"
168+
source = "/var/lib/longhorn"
169+
options = ["bind", "rshared", "rw"]
170+
}
171+
]
172+
}
173+
}
174+
})
175+
]
127176
}
128177

129178
# Apply Worker Configurations

‎longhorn-install.sh‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
#!/bin/bash
2+
# Longhorn Installation Script for Talos Kubernetes Cluster
3+
# This script installs Longhorn and configures storage on all worker nodes
4+
5+
set -e
6+
7+
echo "=== Longhorn Installation for Talos ==="
8+
9+
# Check prerequisites
10+
if ! command -v kubectl &> /dev/null; then
11+
echo "Error: kubectl is not installed"
12+
exit 1
13+
fi
14+
15+
if ! command -v helm &> /dev/null; then
16+
echo "Error: helm is not installed"
17+
exit 1
18+
fi
19+
20+
# Add Longhorn Helm repo
21+
echo "Adding Longhorn Helm repository..."
22+
helm repo add longhorn https://charts.longhorn.io 2>/dev/null || true
23+
helm repo update
24+
25+
# Create namespace with pod security labels
26+
echo "Creating longhorn-system namespace..."
27+
kubectl create namespace longhorn-system 2>/dev/null || true
28+
kubectl label namespace longhorn-system \
29+
pod-security.kubernetes.io/enforce=privileged \
30+
pod-security.kubernetes.io/audit=privileged \
31+
pod-security.kubernetes.io/warn=privileged \
32+
--overwrite
33+
34+
# Install Longhorn
35+
echo "Installing Longhorn..."
36+
helm upgrade --install longhorn longhorn/longhorn \
37+
--namespace longhorn-system \
38+
--set defaultSettings.createDefaultDiskLabeledNodes=true \
39+
--set persistence.defaultClassReplicaCount=3 \
40+
--wait --timeout 5m
41+
42+
# Wait for Longhorn manager pods to be ready
43+
echo "Waiting for Longhorn manager pods..."
44+
kubectl wait --for=condition=ready pod -l app=longhorn-manager -n longhorn-system --timeout=300s
45+
46+
# Expose Longhorn UI via NodePort
47+
echo "Exposing Longhorn UI on NodePort 30080..."
48+
kubectl patch svc longhorn-frontend -n longhorn-system \
49+
-p '{"spec": {"type": "NodePort", "ports": [{"port": 80, "targetPort": 8000, "nodePort": 30080}]}}'
50+
51+
# Get worker nodes
52+
WORKER_NODES=$(kubectl get nodes --no-headers -o custom-columns=":metadata.name" | grep worker)
53+
54+
# Configure disks on each worker node
55+
echo "Configuring storage disks on worker nodes..."
56+
for node in $WORKER_NODES; do
57+
echo " Configuring disk on $node..."
58+
kubectl patch nodes.longhorn.io "$node" -n longhorn-system --type='merge' \
59+
-p '{"spec":{"disks":{"default-disk":{"path":"/var/lib/longhorn","allowScheduling":true,"storageReserved":0}}}}'
60+
done
61+
62+
# Wait for disks to be ready
63+
echo "Waiting for disks to initialize..."
64+
sleep 10
65+
66+
# Verify disk status
67+
echo ""
68+
echo "=== Longhorn Storage Status ==="
69+
for node in $WORKER_NODES; do
70+
STORAGE=$(kubectl get nodes.longhorn.io "$node" -n longhorn-system -o jsonpath='{.status.diskStatus.default-disk.storageAvailable}' 2>/dev/null || echo "0")
71+
STORAGE_GB=$((STORAGE / 1024 / 1024 / 1024))
72+
echo " $node: ${STORAGE_GB} GB available"
73+
done
74+
75+
echo ""
76+
echo "=== Longhorn Installation Complete ==="
77+
echo ""
78+
echo "Longhorn UI available at:"
79+
echo " http://<worker-node-ip>:30080"
80+
echo ""
81+
echo "Worker node IPs:"
82+
kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o wide --no-headers | awk '{print " " $1 ": " $6}'
83+
echo ""
84+
echo "StorageClass 'longhorn' is now the default storage class."

0 commit comments

Comments
 (0)