comfy.rs

A high-performance Rust port of ComfyUI using the Candle ML framework

Bringing the power of ComfyUI workflows to Rust with native performance

---

🎯 Vision

comfy.rs is a ground-up reimplementation of ComfyUI in Rust, leveraging the Candle ML framework. Our goal is to provide a lightweight, fast, and memory-efficient alternative for running stable diffusion and other AI model workflows across all major platforms.

Why comfy.rs?

• 🚀 Performance: 1.5-2x faster inference than Python implementations
• 💾 Memory Efficient: 20-30% lower memory footprint with smart model management
• 📦 Lightweight Deployment: Single binary < 100MB (no Python runtime needed)
• 🔒 Type Safety: Rust's type system catches errors at compile time
• ⚡ Serverless Ready: Fast cold starts, perfect for cloud deployments
• 🌍 Cross-Platform: Native support for Linux, macOS, Windows, and server environments
• 🎮 GPU Acceleration: CUDA, Metal, ROCm support via Candle

---

🏗️ Architecture

Core Components

comfy.rs/
├── comfy-core/           # Core workflow engine
│   ├── graph/           # Node graph representation & execution
│   ├── node/            # Node trait and type system
│   ├── cache/           # Execution caching system
│   └── memory/          # Smart memory management
│
├── comfy-nodes/         # Standard node implementations
│   ├── loaders/         # Model, VAE, CLIP, LoRA loaders
│   ├── samplers/        # KSampler, schedulers, noise generation
│   ├── conditioning/    # CLIP text encode, conditioning ops
│   ├── latent/          # Latent space operations
│   └── image/           # Image processing, VAE encode/decode
│
├── comfy-models/        # Model implementations using Candle
│   ├── clip/            # CLIP models
│   ├── vae/             # VAE implementations
│   ├── unet/            # U-Net diffusion models
│   ├── controlnet/      # ControlNet support
│   └── samplers/        # Sampling algorithms (DPM++, Euler, etc.)
│
├── comfy-server/        # REST API & WebSocket server
│   ├── api/             # HTTP endpoints
│   ├── queue/           # Job queue system
│   ├── websocket/       # Real-time progress updates
│   └── storage/         # Output and cache management
│
├── comfy-cli/           # Command-line interface
│   ├── execute/         # Workflow execution
│   ├── validate/        # Workflow validation
│   └── convert/         # Model conversion utilities
│
└── examples/            # Example workflows and usage
    ├── workflows/       # Sample JSON workflows
    └── tutorials/       # Getting started guides

Key Design Principles

1. Modular Architecture: Each component is independent and reusable
2. Zero-Copy Where Possible: Minimize memory allocations and copies
3. Async-First: Built on Tokio for efficient I/O and parallelism
4. Type-Safe Node System: Strongly-typed inputs/outputs with runtime validation
5. Compatible: Support ComfyUI workflow JSON format
6. Extensible: Plugin system for custom nodes in Rust

---

🗺️ Development Roadmap

Phase 1: Foundation (Weeks 1-4) ✅ Planning

Goals: Establish core architecture and workflow engine

• Project structure and workspace setup
• Core traits definition (Node, Tensor, WorkflowGraph, ExecutionContext)
• JSON workflow parser
• Node graph validation and dependency resolution
• Execution engine with topological sorting
• Basic caching system (content-based hashing)
• Memory management foundation
• Error handling framework

Deliverables:

• Workflow can be loaded from ComfyUI JSON
• Dependency graph is correctly built and validated
• Mock nodes can be executed in correct order

---

Phase 2: Essential Nodes (Weeks 5-10) 🎯 Critical Path

Goals: Implement core nodes for basic txt2img workflow

Milestone 2.1: Model Loading (Weeks 5-6)

• SafeTensors loader integration
• CheckpointLoaderSimple node
  • SD 1.5 support
  • SDXL support
  • SD 3.x support
• VAELoader node
• CLIPLoader node
• Model path configuration system
• Device selection (CPU/CUDA/Metal)

Milestone 2.2: Sampling (Weeks 7-8)

• KSampler node implementation
• Noise generation
• Schedulers:
  • DPM++ 2M
  • DPM++ 2M Karras
  • Euler
  • Euler A
  • LMS
• CFG (Classifier-Free Guidance)
• Latent image tensor handling

Milestone 2.3: Conditioning & Image Processing (Weeks 9-10)

• CLIPTextEncode node
• ConditioningCombine
• ConditioningSetArea
• VAEDecode node
• VAEEncode node
• EmptyLatentImage node
• LoadImage node
• SaveImage node
• PreviewImage node
• ImageScale node

Deliverables:

• Complete txt2img workflow works end-to-end
• Can load any SD1.5/SDXL checkpoint
• Generate images comparable to ComfyUI output
• Basic img2img support

Test Workflow: Standard txt2img (checkpoint → CLIP encode → KSampler → VAE decode → save)

---

Phase 3: Server & API (Weeks 11-14) 🌐 User-Facing

Goals: Production-ready API server

Milestone 3.1: Core Server (Week 11)

• HTTP server with Axum
• REST API endpoints:
  • POST /api/prompt - Queue workflow
  • GET /api/queue - Get queue status
  • GET /api/history - Get execution history
  • DELETE /api/queue/:id - Cancel job
  • POST /api/interrupt - Interrupt current execution
• Job queue system with priorities
• Concurrent execution management

Milestone 3.2: Real-time Updates (Week 12)

• WebSocket server
• Progress events
• Preview image streaming
• Node execution events
• Error notifications

Milestone 3.3: Storage & Management (Week 13)

• Output directory management
• Temporary file cleanup
• Workflow history persistence (SQLite)
• Model cache management
• Configuration file support (TOML/YAML)

Milestone 3.4: CLI Tool (Week 14)

• comfy-rs run <workflow.json> - Execute workflow
• comfy-rs validate <workflow.json> - Validate workflow
• comfy-rs serve - Start API server
• comfy-rs models list - List available models
• comfy-rs info - System information
• Interactive mode for workflow building

Deliverables:

• Production-ready API server
• CLI tool for local execution
• Compatible with ComfyUI frontend (API-compatible)
• Docker image available

---

Phase 4: Advanced Features (Weeks 15-20) 🔥 Power User

Milestone 4.1: LoRA & Advanced Loading (Weeks 15-16)

• LoRALoader node
• Multiple LoRA support
• Strength adjustment
• Hypernetwork support
• Embedding/Textual Inversion
• Model merging nodes

Milestone 4.2: ControlNet (Week 17)

• ControlNet loader
• ControlNetApply node
• Preprocessors:
  • Canny edge detection
  • Depth estimation
  • OpenPose
  • Scribble
  • Segmentation

Milestone 4.3: Upscaling & Image Enhancement (Week 18)

• ImageUpscaleWithModel node
• ESRGAN support
• Real-ESRGAN support
• SwinIR support
• Tiled upscaling for large images
• Batch processing

Milestone 4.4: Advanced Sampling (Week 19)

• Advanced schedulers (DDIM, PLMS, etc.)
• Img2Img with denoising strength
• Inpainting support
• Outpainting
• Area composition (regional prompting)
• IP-Adapter support

Milestone 4.5: Optimizations (Week 20)

• Flash Attention v2 integration
• Quantization support (GGML/GGUF)
• Model offloading strategies
• Mixed precision inference
• Batch processing optimization
• Smart VRAM management
• Memory profiling tools

Deliverables:

• Support for 95% of common ComfyUI workflows
• Performance benchmarks showing improvements
• Production-ready deployment guides

---

Phase 5: Video & Audio (Weeks 21-24) 🎬 Future

• Stable Video Diffusion (SVD)
• AnimateDiff support
• Frame interpolation
• Audio generation models
• Video upscaling
• Batch frame processing

---

Phase 6: Ecosystem (Weeks 25+) 🌟 Community

• Plugin system for custom nodes
• Node development SDK
• Web UI (standalone or integrated)
• Model hub integration (HuggingFace)
• Workflow marketplace
• Documentation site
• Tutorial videos
• Community custom nodes repository

---

🔧 Technical Stack

Core Technologies

Component	Technology	Rationale
ML Framework	Candle	Rust-native, GPU support, lightweight
Async Runtime	Tokio	Industry standard, mature ecosystem
Web Server	Axum	Fast, ergonomic, built on Tokio
Serialization	Serde	JSON workflow parsing
CLI	Clap	Powerful argument parsing
Database	SQLite (via rusqlite)	Embedded, zero-config
Image Processing	image	Pure Rust, format support
Logging	tracing	Structured logging
Testing	Built-in Rust testing + criterion	Benchmarking

Hardware Support

• NVIDIA GPUs: CUDA support via Candle (cuDNN optional)
• AMD GPUs: ROCm support (Linux)
• Apple Silicon: Metal acceleration (M1/M2/M3)
• Intel GPUs: Experimental support
• CPU: Optimized with BLAS (MKL/OpenBLAS/Accelerate)

---

🚀 Getting Started

⚠️ Note: comfy.rs is currently in the planning/early development phase. These instructions are forward-looking.

Prerequisites

• Rust 1.70 or higher
• CUDA Toolkit 11.8+ (for NVIDIA GPU support)
• 8GB+ RAM (16GB+ recommended)

Installation

# Clone the repository
git clone https://github.com/satishbabariya/comfy.rs
cd comfy.rs

# Build all components (CPU only)
cargo build --release

# Build with CUDA support
cargo build --release --features cuda

# Build with Metal support (macOS)
cargo build --release --features metal

Quick Start

# Start the API server
comfy-rs serve --host 0.0.0.0 --port 8188

# Execute a workflow from CLI
comfy-rs run examples/workflows/txt2img.json --output ./outputs

# Validate a workflow
comfy-rs validate my_workflow.json

# List available models
comfy-rs models list

Configuration

Create a config.toml in your working directory:

[paths]
models = ["./models", "~/ComfyUI/models"]
output = "./output"
temp = "./temp"

[server]
host = "127.0.0.1"
port = 8188
max_queue_size = 100

[execution]
default_device = "cuda:0"  # or "cpu", "metal"
max_vram_gb = 10
enable_model_offload = true
cache_size_gb = 4

[performance]
num_threads = 8
enable_flash_attention = true

---

📊 Node System Design

Node Trait

pub trait Node: Send + Sync {
    /// Unique identifier for this node type
    fn node_type(&self) -> &str;
    
    /// Input slot definitions
    fn inputs(&self) -> Vec<InputSlot>;
    
    /// Output slot definitions  
    fn outputs(&self) -> Vec<OutputSlot>;
    
    /// Execute the node
    async fn execute(
        &self,
        ctx: &ExecutionContext,
        inputs: NodeInputs,
    ) -> Result<NodeOutputs>;
    
    /// Validate inputs before execution
    fn validate(&self, inputs: &NodeInputs) -> Result<()>;
}

Type System

pub enum TensorData {
    Image(Tensor),      // [B, C, H, W]
    Latent(Tensor),     // [B, C, H/8, W/8]
    Conditioning(Tensor), // [B, T, D]
    Mask(Tensor),       // [B, 1, H, W]
}

pub enum NodeValue {
    Tensor(TensorData),
    Model(Arc<dyn ModelType>),
    Integer(i64),
    Float(f64),
    String(String),
    Boolean(bool),
    List(Vec<NodeValue>),
}

Workflow Format

comfy.rs uses ComfyUI-compatible JSON format:

{
  "1": {
    "class_type": "CheckpointLoaderSimple",
    "inputs": {
      "ckpt_name": "sd_xl_base_1.0.safetensors"
    }
  },
  "2": {
    "class_type": "CLIPTextEncode",
    "inputs": {
      "text": "a beautiful sunset over mountains",
      "clip": ["1", 0]
    }
  },
  "3": {
    "class_type": "KSampler",
    "inputs": {
      "model": ["1", 0],
      "positive": ["2", 0],
      "negative": ["2", 0],
      "latent_image": ["4", 0],
      "seed": 42,
      "steps": 20,
      "cfg": 7.5,
      "sampler_name": "dpmpp_2m",
      "scheduler": "karras"
    }
  }
}

---

🔬 Performance Targets

Benchmark Goals (vs Python ComfyUI)

Metric	Target	Baseline (Python)
Cold Start	< 3s	~10s
Model Load	< 5s	~8s
SDXL 1024x1024 (20 steps)	< 8s	~12s
SD1.5 512x512 (20 steps)	< 2s	~3.5s
Memory Usage	-25%	100%
Binary Size	< 100MB	~2GB (with Python)

Optimization Strategies

1. Memory:

   • Smart model offloading (GPU ↔ CPU)
   • Aggressive tensor deallocation
   • Memory pooling for tensors
   • Quantization support

2. Speed:

   • Flash Attention v2
   • Fused kernels
   • Parallel node execution
   • Efficient scheduling

3. Deployment:

   • Static binary (no runtime dependencies)
   • Cross-compilation support
   • Minimal Docker images (< 500MB)

---

🤝 Contributing

We welcome contributions! Here's how you can help:

Development Setup

# Clone with submodules
git clone --recursive https://github.com/satishbabariya/comfy.rs

# Run tests
cargo test --all

# Run benchmarks
cargo bench

# Check formatting
cargo fmt --check

# Run linter
cargo clippy -- -D warnings

Adding a New Node

1. Create node implementation in comfy-nodes/src/
2. Implement the Node trait
3. Add tests in tests/nodes/
4. Register in comfy-nodes/src/lib.rs
5. Add example workflow in examples/workflows/

Priority Areas

• Node implementations (see Phase 2)
• Documentation and examples
• Performance optimization
• Test coverage
• Model compatibility testing

---

📈 Project Status

Current Phase: Phase 1 - Planning 📋

Track our progress:

• Project ideation and brainstorming
• Architecture design
• Development roadmap
• Repository setup
• Core traits implementation
• First proof-of-concept node

Compatibility Matrix

Feature	Status	ComfyUI Parity
Workflow Loading	🔴 Not Started	0%
Basic Nodes	🔴 Not Started	0%
SD 1.5	🔴 Not Started	0%
SDXL	🔴 Not Started	0%
LoRA	🔴 Not Started	0%
ControlNet	🔴 Not Started	0%
API Server	🔴 Not Started	0%

Legend: 🔴 Not Started | 🟡 In Progress | 🟢 Complete

---

🎓 Learning Resources

Understanding ComfyUI

• ComfyUI GitHub
• ComfyUI Examples
• Node Documentation

Rust ML & Candle

• Candle Documentation
• Candle Examples
• Stable Diffusion in Candle

Stable Diffusion

• Stable Diffusion Paper
• DDPM Paper
• Understanding Diffusion Models

---

🔮 Future Possibilities

Long-term Vision

• Native UI: Cross-platform desktop app with Tauri/egui
• Cloud Service: Managed hosting for comfy.rs workflows
• WASM Support: Run lightweight models in browsers
• Distributed Execution: Multi-GPU, multi-node execution
• Model Training: Not just inference, but fine-tuning support
• Custom Model Support: Easy integration of new model architectures

Research Areas

• Novel sampling algorithms optimized for Rust
• Advanced caching strategies
• Distributed inference
• On-device mobile deployment

---

📝 License

MIT License - see LICENSE file for details

---

🙏 Acknowledgments

• ComfyUI team for the original implementation and workflow design
• HuggingFace for the Candle framework
• Stability AI for Stable Diffusion models
• The Rust ML community

---

💬 Community & Support

• Discussions: GitHub Discussions
• Issues: GitHub Issues
• Discord: Coming soon
• Twitter/X: Coming soon

---

Built with ❤️ in Rust

Making AI workflows faster, lighter, and more reliable

⭐ Star us on GitHub | 📖 Documentation | 🤝 Contributing