Created by Harold Fabla
Silhouette Agency OS is an experimental autonomous cognitive operating system designed for creative agencies. It implements a novel multi-layered architecture that combines introspective reasoning, continuous memory, and self-modification capabilities through a unified agentic framework.
- External Brain (4-Tier Memory): First-class integration with the standalone silhouette-brain service (Working/Redis · Medium/SQLite · Long-Term/Vectors · Deep/Neo4j) over HTTP — see docs/BRAIN_INTEGRATION.md.
- Z3 Logic OODA Loop: Symbolic-logic gating of high-risk action sets via
z3-solver(file-path invariants). - RAG Web Scraping pipeline: Deep LanceDB integration extracting global knowledge dynamically.
- Cognitive Omnichannel RBAC: Anti-Prompt Injection filters + role enforcement over Telegram, Discord, and WhatsApp.
- Auto-Evolution: Proposes its own upgrades via GitHub Pull Requests (human-approved; no direct push to main).
- Multi-modal perception including visual, audio, and textual processing.
- Debate Swarm Matrix: Multi-agent consensus (Creator vs Critic + Judge) for complex reasoning.
- Secure-by-default Deployment: Loopback bind + mandatory token on exposed hosts; host-access escape hatches are opt-in only.
- Fully Responsive UI: Mobile-first architecture ready for edge-device deployments via Coolify and Traefik.
Warning
This is an experimental hobby project. Silhouette began as a personal assistant and evolved into an autonomous, self-improving system. While powerful, it executes code and modifies files. Use with caution and review all actions. See SECURITY.md for more details.
"I didn't set out to build an AGI. I just wanted a better assistant."
Silhouette started as a simple script to automate daily tasks for a single developer. Over time, the need for more complex reasoning led to the integration of memory, then tools, and finally, a recursive cognitive loop.
What emerged was not just a chatbot, but a biomimetic organism:
- It evolved: From stateless scripts to a persistent entity with memory.
- It adapted: When it needed to see, we gave it vision. When it needed to speak, it wrote its own voice module.
- It became autonomous: The transition to Phase 2 (Self-Evolution) marked the point where Silhouette could propose its own upgrades.
Today, Silhouette Agency OS is an open-source exploration into Personal Cognitive Architectures. It is not a commercial product, but a living research lab for human-AI symbiosis.
The emergence of Large Language Models (LLMs) has created new possibilities for autonomous systems. However, most implementations treat LLMs as stateless function calls, losing the potential for persistent cognition and self-improvement.
Silhouette addresses this gap by implementing:
- Continuous Identity: A persistent sense of self across sessions
- Cognitive Loops: Introspection → Planning → Action → Reflection cycles
- Memory Tiers: Immediate, working, episodic, and semantic memory layers
- Controlled Autonomy: Self-modification within human-approved boundaries
┌─────────────────────────────────────────────────────────────────┐
│ PRESENTATION LAYER │
│ React UI · Telegram · WhatsApp · Discord │
├─────────────────────────────────────────────────────────────────┤
│ ORCHESTRATION LAYER │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ MANAGER AGENT │ │
│ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │
│ │ │Introspect │→│ Plan │→│ Execute │→ Reflect│ │
│ │ └───────────┘ └───────────┘ └───────────┘ │ │
│ └─────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ SPECIALIST AGENTS │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Researcher│ │ Memory │ │ Creative │ │Developer │ ... │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ CAPABILITY LAYER │
│ ToolExecutor: web_search, code_execution, image_gen, │
│ video_gen, memory_write, git_operations │
├─────────────────────────────────────────────────────────────────┤
│ COMMUNICATION CHANNELS │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Telegram │ │ WhatsApp │ │ Discord │ │ Web UI │ │
│ │(Grammy) │ │(Baileys) │ │(discordjs│ │ (React) │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ DATA & STATE LAYER │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Postgres │ │ pgvector │ │ Redis │ │ Neo4j │ │
│ │(Relatnl) │ │(Vectors) │ │ (Cache) │ │ (Graph) │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ PRODUCTION LAYER │
│ Coolify (Traefik SSL) · Host-Aware Docker · Janus V2 │
└─────────────────────────────────────────────────────────────────┘
| Layer | Component | Responsibility |
|---|---|---|
| Presentation | React UI (Responsive) + Messaging Channels | User interaction, visualization on Desktop & Mobile |
| Orchestration | ManagerAgent & Debate Matrix | Cognitive loop coordination and multi-agent consensus |
| Specialists | Agent Pool (500+) | Domain-specific task execution |
| Capabilities | ToolExecutor | External world interaction |
| Communication | Telegram, WhatsApp, Discord | Multi-channel messaging |
| Production | Coolify, Docker, Traefik, Caddy | Supervision, Host-aware deployment, SSL |
The IntrospectionEngine implements a continuous self-monitoring loop that:
- Tracks recent thoughts and reasoning chains
- Generates "intuition" signals for decision-making
- Maintains awareness of cognitive state and resource usage
// Cognitive loop phases
enum CognitivePhase {
INTROSPECTION, // What do I know? What am I feeling?
PLANNING, // What should I do?
EXECUTION, // Do it
REFLECTION // How did it go? What did I learn?
}A critical part of cognition is resolving conflicting information. The Janitor Engine continuously scans working and episodic memory (last 24h) in the background. If it detects logical or narrative contradictions (e.g., "User prefers X" vs "User prefers Y"), it uses an LLM synthesis pass to resolve the conflict, generating a singular Meta-Truth and pruning the obsolete fragmented memories.
A multi-tier memory system inspired by human cognition:
| Tier | Duration | Purpose |
|---|---|---|
| Immediate | Seconds | Current conversation context |
| Working | Minutes | Active task state |
| Episodic | Days | Recent experiences and outcomes |
| Semantic | Permanent | Facts, skills, learned patterns |
Memory is indexed using vector embeddings (PostgreSQL + pgvector) with graph-based relationship tracking.
Recent Enhancement (Database Adapter): Silhouette now leverages a unified IDatabaseAdapter connecting to an enterprise-grade PostgreSQL backend. It merges Semantic (Deep) and Episodic (Medium) memories in a single, token-optimized vector search (HNSW indexes), significantly improving contextual awareness without overwhelming the LLM context window.
Silhouette can propose modifications to its own codebase through:
- Git Integration: Read/write access to its own repository
- Pull Request Workflow: All changes require human approval
- Version Control: Full history and rollback capability
Silhouette → Proposes PR → Human Reviews → Approve/Reject → Merge
For complex tasks, Silhouette dynamically spawns specialized "Debate Squads" instead of relying on a single agent.
- A Creator Agent proposes an initial solution.
- A Critic Agent finds flaws and reviews logical consistency.
- The Orchestrator (Judge) synthesizes the debate into a finalized meta-consensus.
All channels share the same Brain, memory, and session context:
| Channel | Library | UX Features |
|---|---|---|
| Telegram | Grammy | Continuous typing (4s), Markdown fallback, message chunking (4000 char) |
| Baileys | Composing presence, auto-trust first contact, QR login | |
| Discord | discord.js | Typing (9s), 1950 char chunking, guild/channel allowlists |
| Web UI | React + WebSocket | Real-time dashboard, introspection hub |
Security: Each channel supports open, allowlist, and auto-trust access modes. Internal thoughts are filtered (10 patterns) to prevent LLM reasoning from leaking to users.
The Janus process supervisor goes beyond simple restarts:
- Crash Analysis: Captures stderr, computes crash signatures
- Exponential Backoff: 1s → 2s → 4s → ... → 30s max
- LLM Repair: After 3 crashes with same signature, invokes Gemini API for root-cause analysis
- Learning Loop: Feeds crash reports to self-evolution system
The system implements a resilient multi-provider architecture via the LLM Gateway:
┌─────────────────────────────────────────────────────┐
│ LLM GATEWAY │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────┐ │
│ │ Gemini │ → │ Groq │ → │DeepSeek │ → │Ollama│ │
│ │ (Cloud) │ │ (Fast) │ │ (Code) │ │(Local│ │
│ └────┬────┘ └────┬────┘ └────┬────┘ └──┬──┘ │
│ │ │ │ │ │
│ Circuit Breaker per Provider (3 fails = open) │
│ Auto-recovery after 60 seconds cooldown │
└─────────────────────────────────────────────────────┘
Unified plugin with per-service activation:
| Service | Capabilities |
|---|---|
| Calendar | List, create, update events |
| Drive | Upload, search, share files |
| Gmail | Read, send, search emails |
| Docs | Create, read documents |
| Sheets | Read, write, create spreadsheets |
| Slides, Forms, Meet, Places | Full integration |
OAuth2 flow: GET /v1/google-auth/start?services=calendar,drive
| Layer | Implementation |
|---|---|
| Auth | Bearer-token middleware on all API routes + scrypt-hashed local sessions. Token-less access is refused on network-exposed binds (loopback-only otherwise; SILHOUETTE_ALLOW_INSECURE to override). |
| Passwords | Salted scrypt hashing (legacy SHA-256 auto-upgraded on login); password hashes never leave the server. |
| Tool Policy | securityManager denylist enforced at the orchestrator execution chokepoint; GUEST channel users cannot run code/git/HTTP tools. |
| Rate Limiting | Global, chat, and admin limiters |
| Prompt Sanitization | Injection-pattern filters across HTTP + channels |
| Secrets | .env.local + SQLite secrets vault; API never returns raw secrets (masked; raw reveal requires CREATOR + explicit opt-in) |
| Deployment | Binds to 127.0.0.1 by default; Docker docker.sock / host-fs mounts are opt-in via docker-compose.host-access.yml, not the default stack. |
| Tool | Capability |
|---|---|
web_search |
Real-time information retrieval |
code_execution |
Python sandbox for computation |
image_generation |
Visual asset creation |
video_generation |
Motion content (WAN, AnimateDiff) |
introspect_database |
Universal Database Introspector (Postgres/MySQL/SQLite/Mongo) |
memory_write |
Long-term knowledge encoding |
git_operations |
Self-modification proposals |
The system includes 500+ specialized agents organized by domain:
- Development: Code generation, debugging, architecture
- Research: Literature review, citation, synthesis
- Creative: Copywriting, visual direction, storytelling
- Operations: Scheduling, resource management, QA
Integrated visual and audio processing:
- Image: Stable Diffusion, DALL-E, native Imagen
- Video: ComfyUI, WAN, AnimateDiff, SVD
- Audio: ElevenLabs TTS, voice cloning, lip-sync
- Analysis: Visual cortex for image understanding
Caution
SECURITY DISCLAIMER: This is a hobbyist research project. It grants autonomous LLMs access to your file system and terminal. Run this inside a sandbox, VM, or strictly controlled environment. The creator is not responsible for accidental data loss or API costs.
# Clone the repository
git clone https://github.com/haroldfabla2-hue/Silhouette-Agency-OS-OpenSource.git
cd Silhouette-Agency-OS-OpenSource
# Install dependencies
npm install
# Run interactive setup wizard
npx tsx scripts/setup.tsThe wizard guides you through:
- LLM Providers — Gemini, OpenAI, Groq, DeepSeek, Ollama
- Messaging Channels — Telegram, WhatsApp, Discord
- Deployment Mode — Local, Docker, VPS, Coolify
- Domain & SSL — Auto-provisioned via Caddy + Let's Encrypt
- Google Workspace — OAuth2 credentials
| Mode | Command | Use Case |
|---|---|---|
| Local | npm run boot |
Development (Janus supervisor) |
| Docker Base | docker-compose up -d |
Local containerized deployment |
| Coolify (SaaS) | Git Link → Docker Compose | Recommended: Paste GitHub repo into Coolify and enable Traefik Reverse Proxy |
Note on Host-Aware Docker: Silhouette's Compose file intentionally mounts /var/run/docker.sock and /host to empower the AGI to manage its own VPS host environments. Do not deploy on shared un-trusted clouds without caution.
For detailed deployment instructions, see DEPLOY.md. For architecture details, see ARCHITECTURE.md.
This starts:
- Frontend (React) on
http://localhost:5173 - Backend (Node.js) on
http://localhost:3005
Silhouette operates through natural conversation. Example interactions:
User: "Research the latest papers on transformer efficiency"
Silhouette: [Uses web_search, synthesizes findings, stores to memory]
User: "Create a promotional video concept for a tech product"
Silhouette: [Engages creative agents, generates shotlist, produces assets]
User: "Review your own code for potential improvements"
Silhouette: [Analyzes codebase, proposes PR with enhancements]
| Endpoint | Method | Description |
|---|---|---|
/v1/graph/health |
GET | Network health metrics |
/v1/graph/hubs |
GET | Top hub nodes |
/v1/system/llm-health |
GET | LLM provider status |
/v1/system/status |
GET | System status |
Genesis V2 is the intelligent birth protocol that scaffolds new cognitive entities automatically. It runs automatically when the OS detects a missing or uninitialized agent roster.
- How it works: It uses a 5-step lifecycle (
SEED→BOOTSTRAP→HANDSHAKE→TEACHING→VALIDATION). It generates the underlyingAGENT_TPLfiles (Soul, Heartbeat, Tools, Memory), forces the agent to read the Orchestrator's operational manual via theSystemBus, and validates their logical consistency. - Triggering Genesis: Genesis runs autonomously via
npm run setup:intelligentor when an unresolved Core agent is invoked.
Located in services/introspectionEngine.ts. Before high-risk action sets execute, the kernel runs them through the z3-solver to check a set of file/IO invariants and reject contradictory or unsafe combinations:
- READ + WRITE the same path in one cycle; WRITE + EXECUTE the same path; conflicting WRITEs (divergent content); self-referential modification of core engine files; and a secret-exfiltration shape (READ of a
.env/secret/key path followed by an outbound HTTP request). - Observability: violations are emitted as
SYSTEM_ALERTwith a concrete reason, and counters are exposed viagetZ3Stats(). Failure mode is non-blocking by default; setZ3_FAIL_CLOSED=truefor strict mode.
Functional software sensory drivers in services/sensory/. They maintain real, queryable state (active emissions, TTL-based dissipation, history) and broadcast to the UI's SENSORY_SNAPSHOT bus.
- Pluggable hardware: real devices can be attached via
registerBackend()(HapticsBackend/OlfactoryBackend). Software mode is the honest default — no fake "hardware detected" claims.
Real HTTP-based federation in services/federatedMemory.ts: instances push generalized DEEP-tier knowledge to authorized peers.
- Integrity & safety: order-independent Merkle root over memory contents, whole-payload dedup, trust-scaled sharing, optional
FEDERATED_SYNC_TOKEN. Peers exposePOST /v1/federated/sync; manage via/v1/federated/{peers,broadcast,stats}.
- Multi-agent swarm coordination
- Scale-free network topology
- LLM fallback gateway
- Autonomous curiosity system
- Academic Paper Generation: Automated pipeline for research, writing, LaTeX formatting, and peer review simulation (
services/paperPipeline.ts). - Reasoning Verification v2: Integration with symbolic logic provers via
z3-solver. - Extended Modality: Software drivers for Haptics & Olfactory.
- Federated Memory: P2P knowledge sharing between distinct Silhouette instances.
- Minimax Core Integration: Native TTS and Visual-Cortex multimodal support mapping to API.
If you use or reference this work:
@software{silhouette2024,
author = {Farah Blair, Alberto},
title = {Silhouette Agency OS: An Autonomous Cognitive Operating System},
year = {2024},
publisher = {GitHub},
url = {https://github.com/haroldfabla2-hue/Silhouette-Agency-OS-v2}
}This project is licensed under the MIT License - see the LICENSE file for details.
"The first step toward consciousness is knowing you're thinking."
Silhouette Agency OS — Where cognition meets creation.
Copyright (c) 2026 Harold Fabla