Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
682 lines (542 loc) · 22.8 KB

ARCHITECTURE.md

File metadata and controls

682 lines (542 loc) · 22.8 KB
Raw

light-ss-pool Architecture

Overview

light-ss-pool is a pool manager for light-ss instances, enabling management of multiple shadowsocks proxies through subscription URLs, providing unified monitoring, control, and configuration management.

Architecture Decision

Approach: Process-Based Architecture

We spawn light-ss as separate processes rather than importing it as a library or using external process managers.

Rationale

Stability & Isolation:

  • Fault isolation: If one light-ss instance crashes, others continue running
  • Risk mitigation: Memory leaks or crashes in light-ss won't affect pool manager
  • Stability: light-ss may have undiscovered bugs - process isolation protects the pool

Operational Benefits:

  • Debugging: Can inspect, attach debuggers to individual processes using standard tools (ps, htop, gdb, strace)
  • Resource monitoring: Use OS tools to monitor each instance independently
  • Independent updates: Can update light-ss binary without rebuilding light-ss-pool
  • Operational flexibility: Can forcibly kill hung processes, test new versions on subset of instances
  • Different versions: Could even run different light-ss versions simultaneously if needed

Production Proven:

  • Battle-tested approach used by nginx, php-fpm, gunicorn, supervisor, etc.
  • Industry standard for managing pools of worker processes

Trade-offs Accepted

Higher memory overhead:

  • Each process has its own memory space
  • Worth it for the stability and isolation benefits

IPC via REST API:

  • Communication through light-ss management API rather than direct function calls
  • Clean, well-defined interface that's already implemented
  • Slightly higher latency (negligible for management operations)

Slower startup:

  • Process spawning takes longer than goroutine creation
  • Acceptable for proxy pool use case (not hot path)

System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        light-ss-pool                             │
│                                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   REST API   │  │  CLI Client  │  │   Daemon     │          │
│  │   Server     │  │   (lspc)     │  │   Manager    │          │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘          │
│         │                  │                  │                   │
│         └──────────────────┴──────────────────┘                   │
│                            │                                      │
│                   ┌────────▼────────┐                            │
│                   │  Pool Manager   │                            │
│                   │                 │                            │
│                   │  - Instance Map │                            │
│                   │  - Lifecycle    │                            │
│                   │  - Monitor      │                            │
│                   └────────┬────────┘                            │
│                            │                                      │
│         ┌──────────────────┼──────────────────┐                 │
│         │                  │                  │                  │
│    ┌────▼─────┐      ┌────▼─────┐      ┌────▼─────┐           │
│    │Instance 1│      │Instance 2│      │Instance N│           │
│    │          │      │          │      │          │           │
│    │ PID:1001 │      │ PID:1002 │      │ PID:100N │           │
│    └────┬─────┘      └────┬─────┘      └────┬─────┘           │
│         │                  │                  │                  │
└─────────┼──────────────────┼──────────────────┼─────────────────┘
          │                  │                  │
    ┌─────▼──────┐     ┌─────▼──────┐     ┌─────▼──────┐
    │ light-ss   │     │ light-ss   │     │ light-ss   │
    │ Process    │     │ Process    │     │ Process    │
    │            │     │            │     │            │
    │ Port:10801 │     │ Port:10802 │     │ Port:1080N │
    │ API :18090 │     │ API :18091 │     │ API :1809N │
    └────────────┘     └────────────┘     └────────────┘

Component Architecture

Pool Manager (Control Plane)

Responsibilities:

  • Spawn and manage light-ss processes
  • Track instance state (Running, Stopped, Crashed)
  • Monitor instance health via API calls
  • Handle instance lifecycle (start, stop, restart, reload)
  • Coordinate subscription updates
  • Aggregate statistics from all instances

Does NOT:

  • Handle proxy traffic (data plane is in light-ss processes)
  • Directly manage shadowsocks connections
  • Parse proxy protocols (delegated to light-ss)

Instance Wrapper (Process Management)

Responsibilities:

  • Spawn light-ss process with exec.Command
  • Track process PID and state
  • Generate instance-specific config files
  • Communicate with light-ss via REST API
  • Monitor process health
  • Handle graceful shutdown and forced kill
  • Track restart attempts and backoff

Communication:

  • Uses light-ss management API (HTTP/JSON)
  • Health checks: GET /health
  • Statistics: GET /stats
  • Speed test: GET /speedtest
  • Hot reload: POST /reload

Subscription Manager

Responsibilities:

  • Fetch subscription URLs (HTTP GET)
  • Auto-detect format (Base64 vs Clash YAML)
  • Parse shadowsocks configurations
  • Update instance configs in storage
  • Schedule periodic updates
  • Handle update failures gracefully

Does NOT:

  • Automatically restart instances (user must manually restart)
  • Modify running instances (only updates stored config)

Storage Layer

Responsibilities:

  • Persist pool configuration
  • Persist instance configurations
  • Persist subscription definitions
  • Load configuration on pool startup
  • Atomic writes to prevent corruption

Format:

  • YAML files for human readability
  • JSON support for programmatic access
  • File-based (no database dependency)

Process Lifecycle

Instance Startup

1. Pool Manager receives start request
2. Generate unique ID if needed
3. Allocate unique ports (proxy + API)
4. Create instance-specific config file
5. Spawn light-ss process: light-ss start -c <config>
6. Store PID and process handle
7. Wait for health check to pass
8. Mark instance as Running
9. Start health monitoring

Instance Shutdown

1. Pool Manager receives stop request
2. Call light-ss API: POST /stop (graceful)
3. Wait up to 10 seconds for graceful shutdown
4. If still running, send SIGTERM
5. Wait up to 5 seconds
6. If still running, send SIGKILL (forced)
7. Clean up config file (optional)
8. Mark instance as Stopped
9. Stop health monitoring

Instance Crash Handling

1. Health monitor detects failure
2. Mark instance as Crashed
3. Log crash event with context
4. If auto_restart enabled:
   a. Check restart count vs max_retries
   b. Apply backoff delay
   c. Attempt restart
   d. Increment restart count
5. If max retries exceeded:
   a. Mark instance as Failed
   b. Alert operator (log)
   c. Require manual intervention

Port Allocation Strategy

Port Ranges

  • Proxy ports: 10800-11799 (1000 instances max)
  • API ports: 18090-19089 (1000 instances max)

Allocation

  1. Pool maintains port allocation map
  2. When creating instance, find next available port pair
  3. Check for conflicts before spawning process
  4. Release ports when instance is removed
  5. Persist allocations across pool restarts

Configuration Management

Configuration File Locations

Default locations (current working directory):

./config.yaml                # Pool configuration
./data/
├── instances.yaml           # Instance definitions (managed by pool - use CLI only)
├── subscriptions.yaml       # Subscription URLs (managed by pool - use CLI only)
└── instances/               # Per-instance light-ss configs (auto-generated - don't edit)
    ├── instance-001.yaml
    ├── instance-002.yaml
    └── instance-00N.yaml

⚠️ Important: File Management Rules

File Purpose How to Modify
instances.yaml Source of truth for all instances Use CLI only (lspc inst update/add/delete)
subscriptions.yaml Source of truth for subscriptions Use CLI only (lspc subs update/add/delete)
instances/*.yaml Auto-generated light-ss configs Never edit (regenerated automatically)

Why CLI-only?

  • ✅ Pool validates all changes
  • ✅ Maintains consistency between files
  • ✅ Auto-regenerates light-ss configs
  • ✅ Prevents corruption and conflicts

Update Flow:

User: lspc inst update instance-001 --name "New Name"
  ↓
1. Pool validates input
2. Pool updates instances.yaml
3. Pool regenerates data/instances/instance-001.yaml
4. If running: reload or restart instance

Custom locations (via flags):

light-ss-pool start --config /path/to/config.yaml --data-dir /path/to/data

What goes in config.yaml (Pool Configuration):

  • Pool server settings (API listen address, port)
  • Authentication token for pool API
  • light-ss binary path (where to find light-ss executable)
  • Data storage path (where to store instances.yaml, subscriptions.yaml)
  • Monitoring settings (health check interval, auto-restart policy defaults)
  • Logging configuration (level, format, file path)
  • Port allocation ranges (for auto-assigning ports to instances)
  • Daemon settings (PID file location, Unix socket path)

What goes in instances.yaml (Instance Configurations):

  • All shadowsocks server configurations
  • Per-instance proxy ports and API ports
  • Per-instance enable/disable status
  • Per-instance auto-restart settings
  • Instance metadata (name, tags, source)

What goes in subscriptions.yaml (Subscription Definitions):

  • Subscription URLs
  • Update intervals
  • Last update timestamps

Configuration Flow

1. Pool starts, loads config.yaml from:
   - Path specified by --config flag, or
   - ./config.yaml (current directory), or
   - ~/.config/light-ss-pool/config.yaml (fallback)

2. Loads data files from data directory:
   - instances.yaml (instance definitions)
   - subscriptions.yaml (subscription URLs)

3. For each enabled instance:
   a. Generate instance-specific light-ss config from template
   b. Write to data/instances/<id>.yaml
   c. Spawn light-ss with: light-ss start -c data/instances/<id>.yaml

4. Monitor and manage spawned processes

Configuration Priorities

1. Command-line flags (highest priority)
2. Environment variables
3. Config file (config.yaml)
4. Default values (lowest priority)

Example:

# Override config file location
light-ss-pool start --config /etc/light-ss-pool/config.yaml

# Override data directory
light-ss-pool start --data-dir /var/lib/light-ss-pool

# Override API listen address
light-ss-pool start --listen 0.0.0.0:9090

# Override log level
light-ss-pool start --log-level debug

Security Considerations

Process Isolation

  • Each light-ss process runs with same user as pool
  • No privilege separation between instances
  • OS-level process isolation provides security boundary
  • Resource limits can be applied per-process via ulimit/cgroups

API Security

  • Pool API requires bearer token (optional)
  • light-ss instance APIs are localhost-only by default
  • Unix socket option for CLI (better than TCP)
  • Config files should be chmod 600 (contain passwords)

Credential Management

  • Shadowsocks passwords stored in config files
  • Config files not exposed via API
  • Passwords sanitized in API responses
  • No credential caching in memory beyond config

Pool Management

Pool-Level Management APIs

All APIs use /api/ prefix (no versioning).

The pool itself needs management capabilities:

# Pool lifecycle
GET    /api/pool/status               - Pool status and health
GET    /api/pool/health               - Pool health check
GET    /api/pool/version              - Pool version info
GET    /api/pool/stats                - Aggregate statistics
POST   /api/pool/reload               - Reload pool configuration
POST   /api/pool/shutdown             - Graceful shutdown

# Configuration management
GET    /api/pool/config               - Get current pool config (sanitized)
POST   /api/pool/config/update        - Update pool config
POST   /api/pool/config/reload        - Reload config from file
POST   /api/pool/config/validate      - Validate config file

# Bulk operations
POST   /api/inst/start-all       - Start all enabled instances
POST   /api/inst/stop-all        - Stop all running instances
POST   /api/inst/restart-all     - Restart all running instances
POST   /api/inst/speedtest-all   - Speed test all instances

# Logs and diagnostics
GET    /api/pool/logs                 - Get pool logs
GET    /api/pool/debug/info           - Debug information
GET    /api/pool/debug/pids           - All process PIDs
GET    /api/pool/debug/ports          - Port allocations
GET    /api/pool/metrics              - Prometheus metrics (future)

Pool-Level CLI Commands

# Pool status and management
lspc pool status                      - Show pool status
lspc pool reload                      - Reload pool config
lspc pool shutdown                    - Graceful shutdown
lspc pool restart                     - Restart pool daemon (via systemd)

# Configuration management
lspc config show                      - Show current pool config
lspc config edit                      - Edit config file
lspc config reload                    - Reload from file
lspc config validate                  - Validate config file
lspc config path                      - Show config file path

# Logs
lspc logs                             - Show pool logs
lspc logs -f                          - Follow pool logs
lspc logs --instance <id>             - Show instance logs
lspc logs --tail 100                  - Show last 100 lines

# Diagnostics
lspc debug info                       - Show debug information
lspc debug pids                       - Show all process PIDs
lspc debug ports                      - Show port allocations

Instance-Level Management APIs

All instance lifecycle and monitoring operations:

# Instance list and query
GET    /api/inst                 - List all instances
GET    /api/inst?state=running   - List only running instances
GET    /api/inst?enabled=true    - List only enabled instances
GET    /api/inst/:id             - Get instance details
GET    /api/inst/:id/config      - Get instance configuration

# Instance CRUD
POST   /api/inst/add                 - Create instance
POST   /api/inst/:id/update          - Update instance (name, enabled, auto_restart)
POST   /api/inst/:id/delete          - Delete instance
POST   /api/inst/:id/enable          - Enable instance
POST   /api/inst/:id/disable         - Disable instance

# Instance lifecycle - single
POST   /api/inst/:id/start       - Start instance
POST   /api/inst/:id/stop        - Stop instance
POST   /api/inst/:id/restart     - Restart instance
POST   /api/inst/:id/reload      - Hot reload config

# Instance monitoring
GET    /api/inst/:id/health      - Health check instance
GET    /api/inst/:id/stats       - Get instance statistics
GET    /api/inst/:id/logs        - Get instance logs
POST   /api/inst/:id/speedtest   - Run speed test

Instance-Level CLI Commands

# List and query
lspc inst list                    - List all instances
lspc inst list --active           - List running instances only
lspc inst list --enabled          - List enabled instances only
lspc inst show <id>               - Show instance details
lspc inst config <id>             - Get instance configuration

# CRUD operations
lspc inst create -f <file>        - Create new instance
lspc inst update <id> --name <name> - Update instance name
lspc inst delete <id>             - Delete instance
lspc inst enable <id>             - Enable instance
lspc inst disable <id>            - Disable instance

# Lifecycle - single instance
lspc inst start <id>              - Start instance
lspc inst stop <id>               - Stop instance
lspc inst restart <id>            - Restart instance
lspc inst reload <id>             - Hot reload config

# Lifecycle - bulk operations
lspc inst start-all               - Start all enabled instances
lspc inst stop-all                - Stop all running instances
lspc inst restart-all             - Restart all running instances

# Monitoring
lspc inst health <id>             - Check instance health
lspc inst stats <id>              - Get instance statistics
lspc inst logs <id>               - Get instance logs

Subscription Management APIs

# Subscription list and query
GET    /api/subs             - List all subscriptions
GET    /api/subs/:id         - Get subscription details

# Subscription CRUD
POST   /api/subs/add         - Add subscription
POST   /api/subs/:id/update  - Update subscription (name, url, interval)
POST   /api/subs/:id/delete  - Remove subscription
POST   /api/subs/:id/enable  - Enable subscription
POST   /api/subs/:id/disable - Disable subscription

# Subscription operations
POST   /api/subs/:id/fetch   - Fetch subscription now
POST   /api/subs/:id/generate - Generate instances from subscription
POST   /api/subs/fetch-all   - Fetch all subscriptions
POST   /api/subs/generate-all - Generate instances from all

Subscription Management CLI Commands

# List and query
lspc subs list                - List all subscriptions
lspc subs show <id>           - Show subscription details

# CRUD operations
lspc subs add --name <name> --url <url> [--interval <seconds>] - Add subscription
lspc subs update <id> [--name <name>] [--url <url>] [--interval <seconds>] - Update subscription
lspc subs delete <id>         - Remove subscription
lspc subs enable <id>         - Enable subscription
lspc subs disable <id>        - Disable subscription

# Operations
lspc subs fetch <id>          - Fetch specific subscription
lspc subs fetch --all         - Fetch all subscriptions
lspc subs generate <id>       - Generate instances from specific
lspc subs generate --all      - Generate instances from all

Speed Test APIs

# Speed test operations
POST   /api/inst/:id/speedtest   - Run speed test on instance
POST   /api/inst/speedtest-all   - Run speed test on all instances
GET    /api/speedtest/results         - List all speed test results
GET    /api/speedtest/results/:id     - Get speed test result for instance

Speed Test CLI Commands

# Speed test operations
lspc speedtest <id>                   - Test specific instance
lspc speedtest --all                  - Test all instances
lspc speedtest list                   - List all test results
lspc speedtest show <id>              - Show result for instance

Monitoring & Observability

Health Monitoring

Every 30 seconds (configurable):
1. Check process is still running (PID exists)
2. Call light-ss API: GET /health
3. If both pass: instance is healthy
4. If either fails: mark as unhealthy, trigger restart logic

Statistics Collection

Pool aggregates stats from all instances:
- Total/active connections per instance
- Bytes sent/received per instance
- Current speeds (up/down)
- Uptime per instance
- Restart count per instance
- Health status

Speed Testing

On-demand or scheduled:
1. Pool calls instance API: GET /speedtest?duration=10
2. light-ss performs speed test through SS connection
3. Results returned: download speed, latency
4. Pool stores results for comparison
5. Can rank instances by speed for load balancing

Deployment Models

Standalone Daemon

# Start pool in background
light-ss-pool start --daemon

# Pool runs in background with PID file
# Logs to file or syslog
# CLI communicates via REST API

Systemd Service

[Unit]
Description=Light SS Pool Manager
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/light-ss-pool start
Restart=on-failure
User=light-ss-pool

[Install]
WantedBy=multi-user.target

Docker Container (Future)

FROM golang:1.22 AS builder
# Build light-ss and light-ss-pool

FROM alpine:latest
# Copy binaries
# Run pool as PID 1
# Instances run as child processes

Scalability

Current Design

  • Supports up to 1000 instances (port range limit)
  • All instances on same machine
  • Single pool manager process

Future Enhancements

  • Distributed pool across multiple machines
  • Load balancing across instances
  • Automatic failover
  • Geographic distribution
  • Central management API for multiple pools

Failure Modes & Recovery

Pool Manager Crashes

- All light-ss instances continue running
- Pool state lost (in-memory)
- On restart:
  - Load config from files
  - Discover running instances (via PID files or ps)
  - Reconnect to existing processes
  - Resume monitoring

Instance Crashes

- Other instances unaffected
- Pool detects via health check
- Auto-restart if enabled
- User notified via logs
- Stats preserved until restart

Disk Full

- Cannot write new configs
- Existing instances continue running
- New instances cannot start
- Alert operator
- Graceful degradation

Network Issues

- Subscription updates fail (logged)
- Instance API calls timeout (marked unhealthy)
- Proxy traffic unaffected (direct in light-ss)
- Pool continues managing what it can

Design Philosophy

  1. Separation of Concerns: Pool manages processes, light-ss handles proxying
  2. Fail Independently: One instance failure doesn't affect others
  3. Debuggable: Use standard tools, clear logs, visible processes
  4. Recoverable: Pool and instances can restart independently
  5. Observable: Health, stats, logs available via API
  6. Simple: Avoid premature optimization, clear code over clever code
  7. Testable: Process-based makes integration testing straightforward