Skip to content

BLOCKCHAIN_FAILURE_DIAGNOSIS.md

Latest commit

 

History

History
250 lines (190 loc) · 6.28 KB

BLOCKCHAIN_FAILURE_DIAGNOSIS.md

File metadata and controls

250 lines (190 loc) · 6.28 KB
Raw

Blockchain Failure Diagnosis - Hetzner Server

Date: October 28, 2025 Server: 178.156.187.81 (Ashburn, VA)

🔴 Critical Issue Summary

The blockchain on the Hetzner server is completely down. No Docker containers are running, and the build process failed.

📊 Diagnostic Results

✅ Server Status: ONLINE

  • Load: 0.0 (healthy)
  • Memory: 22% usage (750MB/3.7GB) - Healthy
  • Disk: 66.5% usage (50GB/75GB) - Acceptable but approaching warning level
  • Uptime: System restart required (pending updates)

❌ Blockchain Status: DOWN

  • Docker containers: 0 running
  • RPC endpoint: Not responding (ports 26657, 3003)
  • P2P network: Not responding (port 26656)
  • Data directory: Exists but minimal (12KB)

⚠️ System Issues

  • 32 pending system updates
  • System restart required
  • No blockchain containers exist (not stopped, but absent)

🔍 Root Cause Analysis

Primary Cause: Docker Build Failures

The Docker logs show repeated build failures:

Oct 28 01:16:33: exit code: 101 - cargo build --release --bin dytallix-node
Oct 28 01:27:55: rpc error: code = Canceled desc = context canceled
Oct 28 01:43:03: rpc error: code = Canceled desc = context canceled

Analysis:

  1. Cargo build failed - The Rust blockchain node compilation failed with exit code 101
  2. OOM Kill warnings - Memory pressure during build process
  3. Build cancellations - Multiple attempts were canceled (likely due to timeouts or resource constraints)

Secondary Issues:

  1. Insufficient Memory for Build

    • Current: 3.7GB RAM
    • Rust compilation (especially --release builds) requires significant memory
    • No swap configured (0B)

  2. Incomplete Deployment

    • The blockchain was never successfully built/deployed
    • No container artifacts exist
    • No running services for the blockchain core

  3. Missing PM2 Process Manager

    • PM2 is not installed, which was likely expected for process management

🎯 Solutions & Recommendations

Immediate Actions (Choose One)

Option 1: Use Pre-built Docker Image (RECOMMENDED - Fastest)

# SSH into server
ssh root@178.156.187.81

# Navigate to deployment directory
cd /opt/dytallix-fast-launch

# Pull pre-built image instead of building
docker pull dytallix/blockchain-node:latest

# Update docker-compose.yml to use pre-built image
# Then start services
docker-compose up -d

Option 2: Build with More Memory

# Add swap space for build process
ssh root@178.156.187.81 << 'EOF'
  # Create 4GB swap file
  fallocate -l 4G /swapfile
  chmod 600 /swapfile
  mkswap /swapfile
  swapon /swapfile
  echo '/swapfile none swap sw 0 0' >> /etc/fstab
  
  # Now try building again
  cd /opt/dytallix-fast-launch
  docker-compose build
  docker-compose up -d
EOF

Option 3: Build Locally & Push

# Build on your local machine (if it has more resources)
cd /Users/rickglenn/Downloads/dytallix-main/dytallix-fast-launch
docker build -t dytallix-blockchain:latest -f ../dytallix-lean-launch/Dockerfile.node ..

# Tag and push to registry
docker tag dytallix-blockchain:latest your-registry/dytallix-blockchain:latest
docker push your-registry/dytallix-blockchain:latest

# Deploy on Hetzner
ssh root@178.156.187.81 'cd /opt/dytallix-fast-launch && docker-compose pull && docker-compose up -d'

System Improvements

1. Add Swap Space (Critical for low-memory servers)

ssh root@178.156.187.81 << 'EOF'
  fallocate -l 4G /swapfile
  chmod 600 /swapfile
  mkswap /swapfile
  swapon /swapfile
  echo '/swapfile none swap sw 0 0' >> /etc/fstab
EOF

2. Apply System Updates

ssh root@178.156.187.81 'apt-get update && apt-get upgrade -y && reboot'

3. Set Up Auto-Restart

Ensure docker-compose services have restart: unless-stopped or restart: always

4. Add Monitoring

# Install PM2 for process monitoring
ssh root@178.156.187.81 'npm install -g pm2'

# Add Docker container monitoring
ssh root@178.156.187.81 'docker stats --no-stream > /var/log/docker-stats.log'

5. Free Up Disk Space

# Clean up old Docker images and containers
ssh root@178.156.187.81 'docker system prune -a -f'

# Remove old backups
ssh root@178.156.187.81 'du -sh /root/*backup* /opt/*backup*'

📝 Quick Fix Commands

Check if blockchain can be started with existing setup:

ssh root@178.156.187.81 'cd /opt/dytallix-fast-launch && docker-compose up -d'

View build logs:

ssh root@178.156.187.81 'cd /opt/dytallix-fast-launch && docker-compose build 2>&1 | tee build.log'

Monitor Docker logs in real-time:

ssh root@178.156.187.81 'journalctl -u docker -f'

🚨 Prevention & Monitoring

1. Set Up Health Checks

# Add to docker-compose.yml
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3003/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s

2. Enable Auto-Restart

restart: unless-stopped

3. Set Up Alerts

Consider using:

  • Uptime Robot
  • Prometheus + Alertmanager
  • Simple bash script with cron:
*/5 * * * * curl -sf http://178.156.187.81:3003/health || echo "Blockchain down!" | mail -s "Alert" your@email.com

📚 Additional Context

Current Running Services:

  • ✅ dytallix-api.service - API server is running
  • ❌ Blockchain core - Not running
  • ❌ Docker containers - None exist

Server Specs:

  • CPU: 3 vCPU
  • RAM: 4GB (no swap)
  • Disk: 75GB (50GB used)
  • Location: Ashburn, VA

Files to Check:

  • /opt/dytallix-fast-launch/docker-compose.yml - Main config
  • /root/.dytallix/ - Blockchain data (minimal - 12KB)
  • /var/log/dytallix/ - Application logs (if exists)

🎬 Next Steps

  1. Decide on deployment strategy (pre-built vs build on server)
  2. Add swap space (critical for future builds)
  3. Start blockchain services using chosen method
  4. Verify blockchain is running (check RPC endpoint)
  5. Set up monitoring to prevent future downtime
  6. Apply system updates during next maintenance window

📞 Quick SSH Access

ssh root@178.156.187.81

Use the diagnostic script again anytime:

./check-hetzner-blockchain.sh