application-architecture.md

Coolify Application Architecture

Laravel Project Structure

Core Application Directory (app/)

app/
├── Actions/           # Business logic actions (Action pattern)
├── Console/           # Artisan commands
├── Contracts/         # Interface definitions
├── Data/              # Data Transfer Objects (Spatie Laravel Data)
├── Enums/             # Enumeration classes
├── Events/            # Event classes
├── Exceptions/        # Custom exception classes
├── Helpers/           # Utility helper classes
├── Http/              # HTTP layer (Controllers, Middleware, Requests)
├── Jobs/              # Background job classes
├── Listeners/         # Event listeners
├── Livewire/          # Livewire components (Frontend)
├── Models/            # Eloquent models (Domain entities)
├── Notifications/     # Notification classes
├── Policies/          # Authorization policies
├── Providers/         # Service providers
├── Repositories/      # Repository pattern implementations
├── Services/          # Service layer classes
├── Traits/            # Reusable trait classes
└── View/              # View composers and creators

Core Domain Models

Infrastructure Management

Server.php (46KB, 1343 lines)

• Purpose: Physical/virtual server management
• Key Relationships:
  • hasMany(Application::class) - Deployed applications
  • hasMany(StandalonePostgresql::class) - Database instances
  • belongsTo(Team::class) - Team ownership
• Key Features:
  • SSH connection management
  • Resource monitoring
  • Proxy configuration (Traefik/Caddy)
  • Docker daemon interaction

Application.php (74KB, 1734 lines)

• Purpose: Application deployment and management
• Key Relationships:
  • belongsTo(Server::class) - Deployment target
  • belongsTo(Environment::class) - Environment context
  • hasMany(ApplicationDeploymentQueue::class) - Deployment history
• Key Features:
  • Git repository integration
  • Docker build and deployment
  • Environment variable management
  • SSL certificate handling

Service.php (58KB, 1325 lines)

• Purpose: Multi-container service orchestration
• Key Relationships:
  • hasMany(ServiceApplication::class) - Service components
  • hasMany(ServiceDatabase::class) - Service databases
  • belongsTo(Environment::class) - Environment context
• Key Features:
  • Docker Compose generation
  • Service dependency management
  • Health check configuration

Team & Project Organization

Team.php (8.9KB, 308 lines)

• Purpose: Multi-tenant team management
• Key Relationships:
  • hasMany(User::class) - Team members
  • hasMany(Project::class) - Team projects
  • hasMany(Server::class) - Team servers
• Key Features:
  • Resource limits and quotas
  • Team-based access control
  • Subscription management

Project.php (4.3KB, 156 lines)

• Purpose: Project organization and grouping
• Key Relationships:
  • hasMany(Environment::class) - Project environments
  • belongsTo(Team::class) - Team ownership
• Key Features:
  • Environment isolation
  • Resource organization

Environment.php

• Purpose: Environment-specific configuration
• Key Relationships:
  • hasMany(Application::class) - Environment applications
  • hasMany(Service::class) - Environment services
  • belongsTo(Project::class) - Project context

Database Management Models

Standalone Database Models

• StandalonePostgresql.php (11KB, 351 lines)
• StandaloneMysql.php (11KB, 351 lines)
• StandaloneMariadb.php (10KB, 337 lines)
• StandaloneMongodb.php (12KB, 370 lines)
• StandaloneRedis.php (12KB, 394 lines)
• StandaloneKeydb.php (11KB, 347 lines)
• StandaloneDragonfly.php (11KB, 347 lines)
• StandaloneClickhouse.php (10KB, 336 lines)

Common Features:

• Database configuration management
• Backup scheduling and execution
• Connection string generation
• Health monitoring

Configuration & Settings

EnvironmentVariable.php (7.6KB, 219 lines)

• Purpose: Application environment variable management
• Key Features:
  • Encrypted value storage
  • Build-time vs runtime variables
  • Shared variable inheritance

InstanceSettings.php (3.2KB, 124 lines)

• Purpose: Global Coolify instance configuration
• Key Features:
  • FQDN and port configuration
  • Auto-update settings
  • Security configurations

Architectural Patterns

Action Pattern (app/Actions/)

Using lorisleiva/laravel-actions for business logic encapsulation:

// Example Action structure
class DeployApplication extends Action
{
    public function handle(Application $application): void
    {
        // Business logic for deployment
    }
    
    public function asJob(Application $application): void
    {
        // Queue job implementation
    }
}

Key Action Categories:

• Application/: Deployment and management actions
• Database/: Database operations
• Server/: Server management actions
• Service/: Service orchestration actions

Repository Pattern (app/Repositories/)

Data access abstraction layer:

• Encapsulates database queries
• Provides testable data layer
• Abstracts complex query logic

Service Layer (app/Services/)

Business logic services:

• External API integrations
• Complex business operations
• Cross-cutting concerns

Data Flow Architecture

Request Lifecycle

1. HTTP Request → routes/web.php
2. Middleware → Authentication, authorization
3. Livewire Component → app/Livewire/
4. Action/Service → Business logic execution
5. Model/Repository → Data persistence
6. Response → Livewire reactive update

Background Processing

1. Job Dispatch → Queue system (Redis)
2. Job Processing → app/Jobs/
3. Action Execution → Business logic
4. Event Broadcasting → Real-time updates
5. Notification → User feedback

Security Architecture

Multi-Tenant Isolation

// Team-based query scoping
class Application extends Model
{
    public function scopeOwnedByCurrentTeam($query)
    {
        return $query->whereHas('environment.project.team', function ($q) {
            $q->where('id', currentTeam()->id);
        });
    }
}

Authorization Layers

1. Team Membership → User belongs to team
2. Resource Ownership → Resource belongs to team
3. Policy Authorization → app/Policies/
4. Environment Isolation → Project/environment boundaries

Data Protection

• Environment Variables: Encrypted at rest
• SSH Keys: Secure storage and transmission
• API Tokens: Sanctum-based authentication
• Audit Logging: spatie/laravel-activitylog

Configuration Hierarchy

Global Configuration

• InstanceSettings: System-wide settings
• config/: Laravel configuration files

Team Configuration

• Team: Team-specific settings
• ServerSetting: Server configurations

Project Configuration

• ProjectSetting: Project settings
• Environment: Environment variables

Application Configuration

• ApplicationSetting: App-specific settings
• EnvironmentVariable: Runtime configuration

Event-Driven Architecture

Event Broadcasting (app/Events/)

Real-time updates using Laravel Echo and WebSockets:

// Example event structure
class ApplicationDeploymentStarted implements ShouldBroadcast
{
    public function broadcastOn(): array
    {
        return [
            new PrivateChannel("team.{$this->application->team->id}"),
        ];
    }
}

Event Listeners (app/Listeners/)

• Deployment status updates
• Resource monitoring alerts
• Notification dispatching
• Audit log creation

Database Design Patterns

Polymorphic Relationships

// Environment variables can belong to multiple resource types
class EnvironmentVariable extends Model
{
    public function resource(): MorphTo
    {
        return $this->morphTo();
    }
}

Team-Based Soft Scoping

All major resources include team-based query scoping with request-level caching:

// ✅ CORRECT - Use cached methods (request-level cache via once())
$applications = Application::ownedByCurrentTeamCached();
$servers = Server::ownedByCurrentTeamCached();

// ✅ CORRECT - Filter cached collection in memory
$activeServers = Server::ownedByCurrentTeamCached()->where('is_active', true);

// Only use query builder when you need eager loading or fresh data
$projects = Project::ownedByCurrentTeam()->with('environments')->get();

See Database Patterns for full documentation.

Configuration Inheritance

Environment variables cascade from:

1. Shared Variables → Team-wide defaults
2. Project Variables → Project-specific overrides
3. Application Variables → Application-specific values

Integration Patterns

Git Provider Integration

Abstracted git operations supporting:

• GitHub: app/Models/GithubApp.php
• GitLab: app/Models/GitlabApp.php
• Bitbucket: Webhook integration
• Gitea: Self-hosted Git support

Docker Integration

• Container Management: Direct Docker API communication
• Image Building: Dockerfile and Buildpack support
• Network Management: Custom Docker networks
• Volume Management: Persistent storage handling

SSH Communication

• phpseclib/phpseclib: Secure SSH connections
• Multiplexing: Connection pooling for efficiency
• Key Management: PrivateKey model

Testing Architecture

Test Structure (tests/)

tests/
├── Feature/           # Integration tests
├── Unit/              # Unit tests
├── Browser/           # Dusk browser tests
├── Traits/            # Test helper traits
├── Pest.php           # Pest configuration
└── TestCase.php       # Base test case

Testing Patterns

• Feature Tests: Full request lifecycle testing
• Unit Tests: Individual class/method testing
• Browser Tests: End-to-end user workflows
• Database Testing: Factories and seeders

Performance Considerations

Query Optimization

• Eager Loading: Prevent N+1 queries
• Query Scoping: Team-based filtering
• Database Indexing: Optimized for common queries

Caching Strategy

• Redis: Session and cache storage
• Model Caching: Frequently accessed data
• Query Caching: Expensive query results

Background Processing

• Queue Workers: Horizon-managed job processing
• Job Batching: Related job grouping
• Failed Job Handling: Automatic retry logic

Container Status Monitoring System

Overview

Container health status is monitored and updated through multiple independent paths. When modifying status logic, ALL paths must be updated to ensure consistency.

Critical Implementation Locations

1. SSH-Based Status Updates (Scheduled)

File: app/Actions/Docker/GetContainersStatus.php Method: aggregateApplicationStatus() (lines 487-540) Trigger: Scheduled job or manual refresh Frequency: Every minute (via ServerCheckJob)

Status Aggregation Logic:

// Tracks multiple status flags
$hasRunning = false;
$hasRestarting = false;
$hasUnhealthy = false;
$hasUnknown = false;  // ⚠️ CRITICAL: Must track unknown
$hasExited = false;
// ... more states

// Priority: restarting > degraded > running (unhealthy > unknown > healthy)
if ($hasRunning) {
    if ($hasUnhealthy) return 'running (unhealthy)';
    elseif ($hasUnknown) return 'running (unknown)';
    else return 'running (healthy)';
}

2. Sentinel-Based Status Updates (Real-time)

File: app/Jobs/PushServerUpdateJob.php Method: aggregateMultiContainerStatuses() (lines 269-298) Trigger: Sentinel push updates from remote servers Frequency: Every ~30 seconds (real-time)

Status Aggregation Logic:

// ⚠️ MUST match GetContainersStatus logic
$hasRunning = false;
$hasUnhealthy = false;
$hasUnknown = false;  // ⚠️ CRITICAL: Added to fix bug

foreach ($relevantStatuses as $status) {
    if (str($status)->contains('running')) {
        $hasRunning = true;
        if (str($status)->contains('unhealthy')) $hasUnhealthy = true;
        if (str($status)->contains('unknown')) $hasUnknown = true;  // ⚠️ CRITICAL
    }
}

// Priority: unhealthy > unknown > healthy
if ($hasRunning) {
    if ($hasUnhealthy) $aggregatedStatus = 'running (unhealthy)';
    elseif ($hasUnknown) $aggregatedStatus = 'running (unknown)';
    else $aggregatedStatus = 'running (healthy)';
}

3. Multi-Server Status Aggregation

File: app/Actions/Shared/ComplexStatusCheck.php Method: resource() (lines 48-210) Purpose: Aggregates status across multiple servers for applications Used by: Applications with multiple destinations

Key Features:

• Aggregates statuses from main + additional servers
• Handles excluded containers (:excluded suffix)
• Calculates overall application health from all containers

Status Format with Excluded Containers:

// When all containers excluded from health checks:
return 'running:unhealthy:excluded';  // Container running but unhealthy, monitoring disabled
return 'running:unknown:excluded';     // Container running, health unknown, monitoring disabled
return 'running:healthy:excluded';     // Container running and healthy, monitoring disabled
return 'degraded:excluded';            // Some containers down, monitoring disabled
return 'exited:excluded';              // All containers stopped, monitoring disabled

4. Service-Level Status Aggregation

File: app/Models/Service.php Method: complexStatus() (lines 176-288) Purpose: Aggregates status for multi-container services Used by: Docker Compose services

Status Calculation:

// Aggregates status from all service applications and databases
// Handles excluded containers separately
// Returns status with :excluded suffix when all containers excluded
if (!$hasNonExcluded && $complexStatus === null && $complexHealth === null) {
    // All services excluded - calculate from excluded containers
    return "{$excludedStatus}:excluded";
}

Status Flow Diagram

┌─────────────────────────────────────────────────────────────┐
│                    Container Status Sources                  │
└─────────────────────────────────────────────────────────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
┌───────────────┐   ┌─────────────────┐   ┌──────────────┐
│ SSH-Based     │   │ Sentinel-Based  │   │ Multi-Server │
│ (Scheduled)   │   │ (Real-time)     │   │ Aggregation  │
├───────────────┤   ├─────────────────┤   ├──────────────┤
│ ServerCheck   │   │ PushServerUp-   │   │ ComplexStatus│
│ Job           │   │ dateJob         │   │ Check        │
│               │   │                 │   │              │
│ Every ~1min   │   │ Every ~30sec    │   │ On demand    │
└───────┬───────┘   └────────┬────────┘   └──────┬───────┘
        │                    │                    │
        └────────────────────┼────────────────────┘
                             │
                             ▼
                 ┌───────────────────────┐
                 │ Application/Service   │
                 │ Status Property       │
                 └───────────────────────┘
                             │
                             ▼
                 ┌───────────────────────┐
                 │ UI Display (Livewire) │
                 └───────────────────────┘

Status Priority System

All status aggregation locations MUST follow the same priority:

For Running Containers:

1. unhealthy - Container has failing health checks
2. unknown - Container health status cannot be determined
3. healthy - Container is healthy

For Non-Running States:

1. restarting → degraded (unhealthy)
2. running + exited → degraded (unhealthy)
3. dead/removing → degraded (unhealthy)
4. paused → paused
5. created/starting → starting
6. exited → exited (unhealthy)

Excluded Containers

When containers have exclude_from_hc: true flag or restart: no:

Behavior:

• Status is still calculated from container state
• :excluded suffix is appended to indicate monitoring disabled
• UI shows "(Monitoring Disabled)" badge
• Action buttons respect the actual container state

Format: {actual-status}:excluded Examples: running:unknown:excluded, degraded:excluded, exited:excluded

All-Excluded Scenario: When ALL containers are excluded from health checks:

• All three status update paths (PushServerUpdateJob, GetContainersStatus, ComplexStatusCheck) MUST calculate status from excluded containers
• Status is returned with :excluded suffix (e.g., running:healthy:excluded)
• NEVER skip status updates - always calculate from excluded containers
• This ensures consistent status regardless of which update mechanism runs
• Shared logic is in app/Traits/CalculatesExcludedStatus.php

Important Notes for Developers

✅ Container Status Aggregation Service:

The container status aggregation logic is centralized in App\Services\ContainerStatusAggregator.

Status Format Standard:

• Backend/Storage: Colon format (running:healthy, degraded:unhealthy)
• UI/Display: Transform to human format (Running (Healthy), Degraded (Unhealthy))

1. Using the ContainerStatusAggregator Service:

   • Import App\Services\ContainerStatusAggregator in any class needing status aggregation
   • Two methods available:
     • aggregateFromStrings(Collection $statusStrings, int $maxRestartCount = 0) - For pre-formatted status strings
     • aggregateFromContainers(Collection $containers, int $maxRestartCount = 0) - For raw Docker container objects
   • Returns colon format: running:healthy, degraded:unhealthy, etc.
   • Automatically handles crash loop detection via $maxRestartCount parameter

2. State Machine Priority (handled by service):

   • Restarting → degraded:unhealthy (highest priority)
   • Crash loop (exited with restarts) → degraded:unhealthy
   • Mixed state (running + exited) → degraded:unhealthy
   • Running → running:unhealthy / running:unknown / running:healthy
   • Dead/Removing → degraded:unhealthy
   • Paused → paused:unknown
   • Starting/Created → starting:unknown
   • Exited → exited:unhealthy (lowest priority)

3. Test both update paths:

   • Run unit tests: ./vendor/bin/pest tests/Unit/ContainerStatusAggregatorTest.php
   • Run integration tests: ./vendor/bin/pest tests/Unit/
   • Test SSH updates (manual refresh)
   • Test Sentinel updates (wait 30 seconds)

4. Handle excluded containers:

   • All containers excluded (exclude_from_hc: true) - Use CalculatesExcludedStatus trait
   • Mixed excluded/non-excluded containers - Filter then use ContainerStatusAggregator
   • Containers with restart: no - Treated same as exclude_from_hc: true

5. Use shared trait for excluded containers:

   • Import App\Traits\CalculatesExcludedStatus in status calculation classes
   • Use getExcludedContainersFromDockerCompose() to parse exclusions
   • Use calculateExcludedStatus() for full Docker inspect objects (ComplexStatusCheck)
   • Use calculateExcludedStatusFromStrings() for status strings (PushServerUpdateJob, GetContainersStatus)

Related Tests

• tests/Unit/ContainerStatusAggregatorTest.php: Core state machine logic (42 comprehensive tests)
• tests/Unit/ContainerHealthStatusTest.php: Health status aggregation integration
• tests/Unit/PushServerUpdateJobStatusAggregationTest.php: Sentinel update logic
• tests/Unit/ExcludeFromHealthCheckTest.php: Excluded container handling

Common Bugs to Avoid

✅ Prevented by ContainerStatusAggregator Service:

• ❌ Old Bug: Forgetting to track $hasUnknown flag → ✅ Now centralized in service
• ❌ Old Bug: Inconsistent priority across paths → ✅ Single source of truth
• ❌ Old Bug: Forgetting to update all 4 locations → ✅ Only one location to update

Still Relevant:

❌ Bug: Forgetting to filter excluded containers before aggregation ✅ Fix: Always use CalculatesExcludedStatus trait to filter before calling ContainerStatusAggregator

❌ Bug: Not passing $maxRestartCount for crash loop detection ✅ Fix: Calculate max restart count from containers and pass to aggregateFromStrings()/aggregateFromContainers()

❌ Bug: Not handling excluded containers with :excluded suffix ✅ Fix: Check for :excluded suffix in UI logic and button visibility