Local Development with Tilt + kind

This guide explains how to set up and use the BRRTRouter local development environment with Tilt and kind (Kubernetes in Docker).

Overview

The local development setup provides:

Fast iteration: Local Rust builds sync to containers in ~1-2 seconds
Full observability: Prometheus, Grafana, Jaeger, and OpenTelemetry collector
Production-like environment: Run in Kubernetes locally
Hot reload: Automatic rebuild and deployment on code changes
No macOS reliability issues: Service runs reliably in kind cluster

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      kind Cluster                            │
│  ┌────────────────────────────────────────────────────────┐ │
│  │  Namespace: brrtrouter-dev                             │ │
│  │                                                         │ │
│  │  ┌──────────────┐  ┌──────────────┐  ┌─────────────┐ │ │
│  │  │   Pet Store  │  │  Prometheus  │  │   Grafana   │ │ │
│  │  │   :8080      │  │   :9090      │  │   :3000     │ │ │
│  │  └──────────────┘  └──────────────┘  └─────────────┘ │ │
│  │                                                         │ │
│  │  ┌──────────────┐  ┌──────────────┐                   │ │
│  │  │    Jaeger    │  │     OTEL     │                   │ │
│  │  │   :16686     │  │  Collector   │                   │ │
│  │  └──────────────┘  └──────────────┘                   │ │
│  └────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
         ↑                ↑               ↑               ↑
         │                │               │               │
    localhost:8081   localhost:9090  localhost:3000  localhost:16686

Prerequisites

Install the following tools:

macOS

# Docker Desktop
brew install --cask docker

# Kubernetes in Docker
brew install kind

# kubectl
brew install kubectl

# Tilt
brew install tilt

# Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSF https://sh.rustup.rs | sh

Linux

# Docker
# Follow: https://docs.docker.com/engine/install/

# kind
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

# kubectl
# Follow: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

# Tilt
curl -fsSL https://raw.githubusercontent.com/tilt-dev/tilt/master/scripts/install.sh | bash

# Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Quick Start

1. Kind cluster (shared; not created from this repo)

Use the microscaler shared cluster (context kind-kind), same as PriceWhisperer / hauliage:

cd ../shared-kind-cluster   # sibling of BRRTRouter in the monorepo
just dev-up                   # registry + cluster (if missing) + registry-wire + Tilt for platform data

For a standalone BRRTRouter clone without the monorepo:

kind create cluster --config k8s/cluster/kind-config.yaml --wait 120s
kubectl config use-context kind-kind

Start kind-registry (localhost:5001) if it is not already running; just dev-up starts it. Wire nodes after cluster exists (see shared-kind-cluster just registry-wire).

2. Start BRRTRouter Tilt

# From BRRTRouter repo root (starts registry, wires nodes, tilt up)
just dev-up

Or: kubectl config use-context kind-kind then tilt up --port 10353 (10353 avoids the default 10350 used by other Tilts).

Press space in the terminal to open the Tilt web UI.

3. Access Services

Once Tilt reports all services are ready (green checkmarks):

Service	URL	Credentials
Pet Store API	http://localhost:8081	X-API-Key: test123
Grafana	http://localhost:3000	admin / admin
Prometheus	http://localhost:9090	None
Jaeger UI	http://localhost:16686	None

4. Test the API

# Health check
curl http://localhost:8081/health

# List pets (requires API key)
curl -H "X-API-Key: test123" http://localhost:8081/pets

# Run all curl tests
just curls

5. Stop Development Environment

# Stop Tilt (Ctrl+C in Tilt terminal, or:)
tilt down

# Tear down kind cluster
./scripts/dev-teardown.sh

# Or use justfile
just dev-down

Development Workflow

Fast Iteration Cycle

Edit Rust source code in src/ or examples/pet_store/src/
Save the file
Tilt automatically:
- Runs cargo build --release locally (fast incremental compilation)
- Syncs the binary to the container (~1-2 seconds)
- Restarts the service
Test immediately at http://localhost:8081

Regenerate from OpenAPI Spec

# Option 1: Automatic (Tilt watches templates/ and examples/openapi.yaml)
# Just edit the spec or templates and save

# Option 2: Manual trigger
# In Tilt UI, click the "regenerate-petstore" button

# Option 3: Command line
cargo run --bin brrtrouter-gen -- generate --spec examples/openapi.yaml --force

Run Tests

# In Tilt UI, click "run-curl-tests" button

# Or from command line:
just curls

# Run Goose load test
cargo run --release --example api_load_test -- \
  --host http://localhost:8081 \
  --users 10 \
  --hatch-rate 2 \
  --run-time 30s

View Logs

Logs are streaming in real-time in the Tilt UI. You can also:

# View petstore logs
kubectl logs -n brrtrouter-dev -l app=petstore -f

# View all pods
kubectl get pods -n brrtrouter-dev

# Describe petstore deployment
kubectl describe deployment -n brrtrouter-dev petstore

Observability

Metrics (Prometheus)

Open http://localhost:9090

Try queries:

# Request rate
rate(brrtrouter_requests_total[5m])

# Response latency
histogram_quantile(0.95, rate(brrtrouter_request_duration_seconds_bucket[5m]))

# Error rate
rate(brrtrouter_errors_total[5m])

Dashboards (Grafana)

Open http://localhost:3000
Login with admin / admin
Navigate to Dashboards → BRRTRouter Pet Store
See real-time metrics:
- Request rate
- Response latency (p50, p95)
- Error rate
- Active connections

Traces (Jaeger)

Open http://localhost:16686
Select service: petstore
Click Find Traces
Explore distributed traces showing:
- Request flow
- Timing breakdown
- Span details

Troubleshooting

Port Already in Use

If you see "port 8080 already in use":

# Find and kill process using port 8080
lsof -ti:8080 | xargs kill -9

# Or use a different port (edit k8s/cluster/kind-config.yaml)

kind Cluster Won't Start

Cluster lifecycle (default name kind, context kind-kind) is owned by the public shared-kind-cluster repository — not by BRRTRouter. Delete and recreate from that repo so the registry, kind-config.yaml, and registry-wire stay consistent (MIT-licensed; see that repo’s LICENSE).

Monorepo (sibling layout):

cd ../shared-kind-cluster
just cluster-delete    # removes the cluster; does not stop kind-registry
just dev-up            # registry → create cluster if missing → registry-wire → Tilt (platform data + observability, UI 10348)

Then return to BRRTRouter and run just dev-up (or tilt up --port 10353).

Standalone clone: Clone shared-kind-cluster next to BRRTRouter, run the same commands from that directory, then use BRRTRouter’s just dev-up. See the shared repo README for prerequisites and layout.

Tilt Build Fails

# Check Rust compilation locally first
cargo check --all

# Clean and rebuild
cargo clean
tilt down --port 10353
tilt up --port 10353

Linker: `ProcessFdQuotaExceeded` (macOS)

Large Rust links (and tools like cargo-zigbuild) open many .rlib archives at once. The default per-process open-file limit can be too low, so the linker fails with ProcessFdQuotaExceeded.

Per terminal (immediate):

ulimit -n 65536
# then rebuild, e.g. cargo build or your Tilt workflow

System-wide (LaunchDaemon / launchctl limit maxfiles on macOS): After raising kernel or launchd limits, restart the Mac or log out and back in so new limits apply reliably to GUI apps, Terminal, and Rust-analyzer—not only shells where you ran ulimit.

Service Not Responding

# Check pod status
kubectl get pods -n brrtrouter-dev

# Check pod logs
kubectl logs -n brrtrouter-dev -l app=petstore

# Restart deployment
kubectl rollout restart deployment/petstore -n brrtrouter-dev

Slow Builds

# Ensure you're using release builds (faster runtime)
cargo build --release

# Check Tilt is using live_update (not full rebuilds)
# Look for "Syncing files" in Tilt UI logs

# Clear Tilt cache
rm -rf .tilt-cache/

Advanced

Customize Resource Limits

Edit k8s/petstore-deployment.yaml:

resources:
  requests:
    memory: "256Mi"  # Increase for larger workloads
    cpu: "200m"
  limits:
    memory: "1Gi"
    cpu: "2000m"

Add Custom Middleware

Edit src/middleware/mod.rs
Save (Tilt rebuilds automatically)
Binary syncs to container
Service restarts
Test immediately

Debug with `RUST_LOG`

Edit k8s/petstore-deployment.yaml:

env:
  - name: RUST_LOG
    value: "trace"  # or debug, info, warn, error

Apply changes:

kubectl apply -f k8s/petstore-deployment.yaml

Multiple Environments

Run multiple kind clusters:

# Edit k8s/cluster/kind-config.yaml and change cluster name
name: brrtrouter-staging

# Create with different name
kind create cluster --config k8s/cluster/kind-config.yaml

# Switch contexts
kubectl config use-context kind-brrtrouter-staging

Comparison: Tilt vs cargo run

Feature	Tilt + kind	cargo run
Reliability on macOS	✅ Always works	⚠️ Intermittent issues
Observability	✅ Full stack included	❌ Manual setup
Hot Reload	✅ ~1-2 seconds	⚠️ Full restart
Production Parity	✅ Kubernetes	❌ Local process
Setup Time	~2 minutes	~10 seconds
Resource Usage	~2GB RAM	~200MB RAM

Tips and Best Practices

Use Tilt UI: Press space after tilt up to open the web interface
Watch Logs: Keep Tilt UI open to see real-time logs and build status
Manual Triggers: Use buttons in Tilt UI for on-demand actions
Resource Labels: Tilt organizes resources by labels (build, app, observability, tools)
Parallel Builds: Library builds happen in parallel with Docker operations
Clean Slate: Run just dev-down && just dev-up for a fresh environment

Next Steps

Getting Help

If you encounter issues:

Check Tilt UI logs for error messages
Run kubectl get pods -n brrtrouter-dev to see pod status
Check this troubleshooting guide
Open an issue on GitHub with:
- Tilt logs
- kubectl describe pod <pod-name> -n brrtrouter-dev output
- Your OS and tool versions

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local Development with Tilt + kind

Overview

Architecture

Prerequisites

macOS

Linux

Quick Start

1. Kind cluster (shared; not created from this repo)

2. Start BRRTRouter Tilt

3. Access Services

4. Test the API

5. Stop Development Environment

Development Workflow

Fast Iteration Cycle

Regenerate from OpenAPI Spec

Run Tests

View Logs

Observability

Metrics (Prometheus)

Dashboards (Grafana)

Traces (Jaeger)

Troubleshooting

Port Already in Use

kind Cluster Won't Start

Tilt Build Fails

Linker: `ProcessFdQuotaExceeded` (macOS)

Service Not Responding

Slow Builds

Advanced

Customize Resource Limits

Add Custom Middleware

Debug with `RUST_LOG`

Multiple Environments

Comparison: Tilt vs cargo run

Tips and Best Practices

Next Steps

Getting Help

FilesExpand file tree

LOCAL_DEVELOPMENT.md

Latest commit

History

LOCAL_DEVELOPMENT.md

File metadata and controls

Local Development with Tilt + kind

Overview

Architecture

Prerequisites

macOS

Linux

Quick Start

1. Kind cluster (shared; not created from this repo)

2. Start BRRTRouter Tilt

3. Access Services

4. Test the API

5. Stop Development Environment

Development Workflow

Fast Iteration Cycle

Regenerate from OpenAPI Spec

Run Tests

View Logs

Observability

Metrics (Prometheus)

Dashboards (Grafana)

Traces (Jaeger)

Troubleshooting

Port Already in Use

kind Cluster Won't Start

Tilt Build Fails

Linker: ProcessFdQuotaExceeded (macOS)

Service Not Responding

Slow Builds

Advanced

Customize Resource Limits

Add Custom Middleware

Debug with RUST_LOG

Multiple Environments

Comparison: Tilt vs cargo run

Tips and Best Practices

Next Steps

Getting Help

Linker: `ProcessFdQuotaExceeded` (macOS)

Debug with `RUST_LOG`