Merge pull request #17 from FuzzingLabs/docs/update-temporal-architecture

docs: Update documentation for v0.7.0 Temporal architecture

Author: tduhamel42

docs/blog/2025-01-16-v0.7.0-temporal-workers-release.md

@@ -0,0 +1,321 @@
+# FuzzForge v0.7.0: Temporal Orchestration & Vertical Workers Architecture
+
+We're excited to announce **FuzzForge v0.7.0**, a major release featuring two significant improvements:
+
+1. **Architectural Foundation**: Complete migration from Prefect to **Temporal** orchestration with **vertical workers** - pre-built containers for instant deployment
+2. **AI-Powered Secret Detection**: New workflows achieving 84% recall on obfuscated secrets using LLM semantic analysis
+
+This release transforms how security workflows are built, deployed, and scaled.
+
+<!-- truncate -->
+
+## 🚀 Flagship Features
+
+### Temporal Orchestration: Production-Ready Workflow Engine
+
+We've fully migrated from Prefect to [Temporal](https://temporal.io), bringing enterprise-grade workflow orchestration to FuzzForge:
+
+**Why Temporal?**
+
+- ✅ **Reliability**: Automatic retries, timeouts, and failure handling built-in
+- ✅ **Observability**: World-class UI for monitoring workflow execution, logs, and debugging
+- ✅ **Scalability**: Horizontal scaling across workers with intelligent load balancing
+- ✅ **Developer Experience**: Type-safe workflows, versioning, and zero downtime deployments
+
+**What This Means for You:**
+
+```bash
+# Start FuzzForge with Temporal
+docker compose up -d
+
+# Monitor workflows in real-time
+open http://localhost:8080  # Temporal UI
+
+# Submit workflows - everything just works
+cd your_project/
+ff workflow run security_assessment .
+```
+
+The Temporal UI gives you complete visibility into workflow execution:
+
+- Live activity timelines
+- Detailed logs for every step
+- Retry history and failure analysis
+- Performance metrics and bottleneck detection
+
+### Vertical Workers: Pre-Built Security Toolchains
+
+FuzzForge now uses **vertical workers** - long-lived containers pre-built with security toolchains for different languages and platforms:
+
+| Worker      | Toolchain                     | Status        | Available Workflows                   |
+| ----------- | ----------------------------- | ------------- | ------------------------------------- |
+| **python**  | Gitleaks, TruffleHog, Atheris | ✅ Production | Secret detection, security assessment |
+| **rust**    | cargo-fuzz                    | ⚠️ Early Dev  | Rust fuzzing                          |
+| **ossfuzz** | OSS-Fuzz infrastructure       | ⚠️ Heavy Dev  | Continuous fuzzing campaigns          |
+
+**Note:** Additional workers (web, android, Go) are planned but not yet available.
+
+**Key Benefits:**
+
+1. **Zero Build Time**: Workflows start instantly - no container builds per workflow
+2. **Instant Code Changes**: Modify workflow code, restart worker, done
+3. **Consistent Environment**: Same toolchain versions across all runs
+4. **Resource Efficiency**: Share workers across multiple concurrent workflows
+
+**Example: Running Secret Detection**
+
+```bash
+# Worker is already running with Gitleaks, TruffleHog installed
+ff workflow run gitleaks_detection .
+
+# Behind the scenes:
+# 1. CLI uploads project to MinIO
+# 2. Temporal schedules on python-worker
+# 3. Worker downloads from MinIO
+# 4. Gitleaks runs (already installed!)
+# 5. Results returned as SARIF
+```
+
+### MinIO Storage: Unified File Handling
+
+We've replaced volume mounts with **MinIO** (S3-compatible object storage):
+
+**Old Way (Volume Mounts):**
+
+```yaml
+# Had to mount directories, manage paths, cleanup manually
+volumes:
+  - ./my_project:/target
+```
+
+**New Way (MinIO):**
+
+```bash
+# CLI handles everything automatically
+ff workflow run security_assessment .
+# ✓ Creates tarball
+# ✓ Uploads to MinIO
+# ✓ Passes target_id to workflow
+# ✓ Worker downloads and extracts
+# ✓ Cleanup handled automatically
+```
+
+**Benefits:**
+
+- ✅ No path conflicts or permissions issues
+- ✅ Works seamlessly with remote Temporal clusters
+- ✅ Automatic cleanup and caching
+- ✅ Supports large targets (GB+)
+
+## 🔍 AI-Powered Secret Detection: Also in v0.7.0
+
+Alongside the architectural improvements, we're releasing a comprehensive **secret detection** system with three workflows:
+
+### Benchmark Results
+
+We tested on a controlled dataset of **32 documented secrets** (12 Easy, 10 Medium, 10 Hard):
+
+| Tool                  | Recall    | Secrets Found | Speed | Best For                    |
+| --------------------- | --------- | ------------- | ----- | --------------------------- |
+| **LLM (gpt-5-mini)**  | **84.4%** | 41            | 618s  | Obfuscated & hidden secrets |
+| **LLM (gpt-4o-mini)** | 56.2%     | 30            | 297s  | Balanced speed/accuracy     |
+| **Gitleaks**          | 37.5%     | 12            | 5s    | Fast pattern-based scanning |
+| **TruffleHog**        | 0.0%      | 1             | 5s    | Entropy analysis            |
+
+📊 [Full benchmark methodology and results →](https://github.com/FuzzingLabs/fuzzforge_ai/blob/dev/backend/benchmarks/by_category/secret_detection/results/comparison_report.md)
+
+### Why LLM-Based Detection Wins
+
+**Obfuscated Secrets (Medium Difficulty):**
+
+```python
+# Gitleaks: ❌ Missed (no pattern match)
+# LLM: ✅ Found (semantic understanding)
+aws_key = base64.b64decode("QUtJQUlPU0ZPRE5ON0VYQU1QTEU=").decode()
+```
+
+**Well-Hidden Secrets (Hard Difficulty):**
+
+```python
+# Gitleaks: ❌ Missed (no pattern)
+# LLM: ✅ Found (understands XOR + join)
+secret = ''.join(chr(ord(c) ^ 0x42) for c in "\x0b\x15\x04\x1b...")
+```
+
+**Standard Secrets (Easy Difficulty):**
+
+```python
+# Both find these:
+AWS_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
+```
+
+### Try It Yourself
+
+```bash
+# Start FuzzForge
+docker compose up -d
+
+# Run secret detection on your code
+cd your_project/
+ff workflow run gitleaks_detection .      # Fast pattern-based
+ff workflow run trufflehog_detection .    # Entropy analysis
+ff workflow run llm_secret_detection .    # AI semantic analysis
+
+# Get SARIF output
+ff finding <run-id>
+```
+
+## 📊 Real-World Impact
+
+**Before v0.7.0 (Pattern-Only Detection):**
+
+- Found: Standard API keys, simple patterns
+- Missed: Base64-encoded secrets, obfuscated credentials, split secrets
+
+**After v0.7.0 (LLM + Patterns):**
+
+- **84% recall** on comprehensive benchmark
+- Detects novel obfuscation techniques
+- Understands code context (not just regex)
+- Catches secrets in:
+  - Base64/hex encoding
+  - String concatenation
+  - XOR/ROT13 obfuscation
+  - Template strings
+  - Binary literals
+
+## 🔄 Migration Guide
+
+### What Changed
+
+**Docker Compose:**
+
+```bash
+# Old (Prefect)
+docker-compose up
+
+# New (Temporal)
+docker compose up -d
+```
+
+**Workflow Submission:**
+
+```bash
+# Old (volume mounts)
+ff workflow run security_assessment --volume ./project
+
+# New (automatic upload)
+ff workflow run security_assessment .
+# CLI handles upload automatically!
+```
+
+**Worker Management:**
+
+```bash
+# Old (per-workflow containers)
+# Each workflow built its own container
+
+# New (vertical workers)
+docker compose up -d  # All workers start
+# Workflows share workers - much faster!
+```
+
+### Configuration
+
+Set up AI workflows with API keys:
+
+```bash
+cp volumes/env/.env.example volumes/env/.env
+# Edit .env and add your API keys (OpenAI, Anthropic, etc.)
+```
+
+Required for:
+
+- `llm_secret_detection` workflow
+- AI agent features (`ff ai agent`)
+
+Basic security workflows (gitleaks, trufflehog, security_assessment) work without this.
+
+## 🏗️ Architecture Overview
+
+```
+┌─────────────┐
+│   User CLI  │ Upload → MinIO
+└──────┬──────┘
+       ↓ Submit
+┌─────────────┐
+│  Temporal   │ Schedule → Task Queue
+└──────┬──────┘
+       ↓ Execute
+┌─────────────┐
+│  Vertical   │ Download from MinIO → Run Tools → Upload Results
+│  Workers    │
+└─────────────┘
+  rust, python, web, android, ossfuzz
+```
+
+**Benefits:**
+
+- 🔄 Automatic retries and timeouts (Temporal)
+- 📦 No file path management (MinIO)
+- ⚡ Zero container build time (Vertical Workers)
+- 📈 Horizontal scaling ready (Temporal + Workers)
+
+## 🎯 Workflow Stability Status
+
+### ✅ Stable & Production-Ready
+
+- **Secret Detection**: `gitleaks_detection`, `trufflehog_detection`, `llm_secret_detection`
+- **Security Assessment**: `security_assessment`
+- Temporal orchestration with python worker
+- MinIO file storage
+
+### ⚠️ Early Development (Functional but not production-ready)
+
+- **Fuzzing workflows**:
+  - `atheris_fuzzing` - Python fuzzing with Atheris
+  - `cargo_fuzzing` - Rust fuzzing with cargo-fuzz
+- **OSS-Fuzz integration**: `ossfuzz_campaign` (under heavy active development)
+
+**Important:** Fuzzing workflows are functional for testing but not recommended for production use yet.
+
+## 📚 Resources
+
+- 🌐 [Website](https://fuzzforge.ai)
+- 📖 [Documentation](https://docs.fuzzforge.ai)
+- 💬 [Discord Community](https://discord.gg/8XEX33UUwZ)
+- 🎓 [FuzzingLabs Academy](https://academy.fuzzinglabs.com/?coupon=GITHUB_FUZZFORGE)
+- 📊 [Secret Detection Benchmarks](https://github.com/FuzzingLabs/fuzzforge_ai/blob/dev/backend/benchmarks/by_category/secret_detection/results/comparison_report.md)
+
+## 🙏 Acknowledgments
+
+Special thanks to:
+
+- [Temporal](https://temporal.io) for the amazing workflow engine
+- Our community for feedback during the migration
+
+## 🚀 Get Started
+
+```bash
+# Clone and install
+git clone https://github.com/fuzzinglabs/fuzzforge_ai.git
+cd fuzzforge_ai
+uv tool install --python python3.12 .
+
+# Start FuzzForge with Temporal
+docker compose up -d
+
+# Run your first workflow
+cd test_projects/vulnerable_app/
+fuzzforge init
+ff workflow run security_assessment .
+
+# Check Temporal UI
+open http://localhost:8080
+```
+
+---
+
+**FuzzForge v0.7.0** is a foundational release that sets the stage for scalable, production-ready security automation. Try it today and let us know what you think!
+
+Star us on [GitHub](https://github.com/FuzzingLabs/fuzzforge_ai) ⭐

docs/docs/ai/prompts.md

@@ -20,7 +20,7 @@ Use the `fuzzforge ai agent` shell to mix structured slash commands with natural
 You> list available fuzzforge workflows
 Assistant> [returns workflow names, descriptions, and required parameters]
 
-You> run fuzzforge workflow static_analysis_scan on ./backend with target_branch=main
+You> run fuzzforge workflow security_assessment on ./backend
 Assistant> Submits the run, emits TaskStatusUpdateEvent entries, and links the SARIF artifact when complete.
 
 You> show findings for that run once it finishes

docs/docs/concept/architecture.md

@@ -52,14 +52,14 @@ graph TB
     end
 
     subgraph "Execution Layer"
-        Docker[Docker Engine]
-        Containers[Workflow Containers]
-        Registry[Docker Registry]
+        VerticalWorkers[Vertical Worker Containers]
+        Tools[Pre-installed Toolchains]
+        WorkerCache[Worker Cache /cache]
     end
 
     subgraph "Storage Layer"
         PostgreSQL[PostgreSQL Database]
-        Volumes[Docker Volumes]
+        MinIO[MinIO S3 Storage]
         Cache[Result Cache]
     end
 
@@ -73,15 +73,15 @@ graph TB
 
     Temporal --> Workers
     Workers --> Scheduler
-    Scheduler --> Docker
+    Scheduler --> VerticalWorkers
 
-    Docker --> Containers
-    Docker --> Registry
-    Containers --> Volumes
+    VerticalWorkers --> Tools
+    VerticalWorkers --> WorkerCache
+    VerticalWorkers --> MinIO
 
     FastAPI --> PostgreSQL
     Workers --> PostgreSQL
-    Containers --> Cache
+    FastAPI --> MinIO
 ```
 
 ## What Are the Main Components?
@@ -201,7 +201,6 @@ services:
 
 Example configuration:
 ```bash
-COMPOSE_PROJECT_NAME=fuzzforge
 DATABASE_URL=postgresql://postgres:postgres@postgres:5432/fuzzforge
 TEMPORAL_ADDRESS=temporal:7233
 S3_ENDPOINT=http://minio:9000

docs/docs/concept/docker-containers.md

@@ -141,7 +141,7 @@ Example build:
 cd workers/rust
 docker build -t fuzzforge-worker-rust:latest .
 # Or via docker-compose
-docker-compose -f docker-compose.temporal.yaml build worker-rust
+docker-compose -f docker-compose.yml build worker-rust
 ```
 
 ---