Skip to content

planning.md

Latest commit

 

History

History
638 lines (471 loc) · 22.2 KB

planning.md

File metadata and controls

638 lines (471 loc) · 22.2 KB
Raw

AI-6 Project

This is going to be an agentic AI project with one or more UIs. Slack is going to be one of the UIs. It is focused on tool use and management.

Roadmap

Research

Look into the Google ecosystem for LLMs and tools.

  • Gemini 3 Pro is arguably the best LLM available today.
  • Look into their OpenAI compatibility layer
  • If it doesn't work then build a native Gemini LLM provider
  • Look into context optimization (codebooks, deltas for editing operations, smart per request context management, sub-agents, etc)

On the OpenAI side, look into the following:

For voice in general, ElvenLabs is highly recommended, but only 10K credits per month, which are you good for about 15 minutes of conversational AI. https://elevenlabs.io

They have a free plan of 10-15 minutes per month, which is good for testing.

Check out DevSpace for hot reloading k8s workloads

Software Development Life Cycle

  • Saar and Gigi will create PRs and merge them at will. Review is optional.
  • Use Github Actions for CI/CD

Capabilities

  • Streaming
  • Context window management (summarization)
  • Long term memory (On startup AI-6 reads context from persistent storage like file or DB, periodically checkpoints)
  • Expose usage information (tokens)
  • Configuration
  • MCP support (engine is MCP client, local tools can run as MCP server)
  • A2A support
  • Async tool use (continue interacting with the user while tools are running in the background)
  • Switch to uv
  • Switch AI-6 core to async
  • Custom AI-6 frontend from scratch (web-based, mobile-friendly)
  • AI2UI integration (https://github.com/google/A2UI/)
  • MCP Apps (https://github.com/modelcontextprotocol/ext-apps)
  • Tool dependency injection (support tools that require constructor arguments like engine, config, etc.)
  • Add pipe support (e.g. ls | grep foo) to CommandTool
  • Parallel tool execution (run multiple tools in parallel and wait for all of them to finish)
  • Add extensible ContextManager responsible for compacting the session history using different strategies
  • REST API (for the engine)
  • GraphQL frontend with Apollo connectors talking to the engine's REST API using Apollo MCP server
  • Graceful handling of rate-limiting
  • Dynamic model selection (e.g. use a different model for different tasks)
  • Computer use (browser and debugging in the IDE!)
  • Voice UI

Tools

  • Kubectl
  • Github / Github Actions
  • AWS
  • AI-6 (recursive agent mesh)
  • Slack tool
  • dOpus integration (track and schedule music listening)
  • Cloudflare

Tool Discovery System

Current Implementation:

  • Auto-discovery scans tool directories for Tool subclasses
  • Assumes all discoverable tools have no-argument constructors (tool = ToolClass())
  • Memory tools (ListSessions, etc.) are discovered with engine=None but manually overridden with proper engine reference
  • Base classes (MCPTool, CommandTool) are explicitly skipped to avoid instantiation errors

Limitations:

  • Tools requiring constructor arguments (dependencies) cannot be auto-discovered
  • Circular dependency: engine needs tools, but tools need engine reference
  • Memory tools are discovered twice (broken + working versions)
  • No support for tools requiring configuration, user parameters, etc.

Future Enhancement - Dependency Injection:

  • Two-phase initialization: create engine core → discover tools with DI
  • Use inspection to detect constructor dependencies (engine, config, user, etc.)
  • Factory pattern with dependency providers
  • Would enable proper support for tools requiring arguments while maintaining auto-discovery

Permission model

  • [] Document the security model
    • OS user based, access to remote services credentials and k8s clusters, run in a container
  • [] Dedicated tool support for defining OS model and permissions for AI-6 and specific tools
  • [] Run in a container (mounting directories and config files like .kube/config and .aws/config)

Sibling Projects

Putting it here for now, so we have a one-stop shop for planning. The Brain and Issue Manager can be utilized by AI-6. The Claudenetes project may or may not have synergy with AI-6.

Brain

Continuous learning for agents. Hierarchical memory system with pluggable backend for different types of memory:

  • session memory (in-memory + complete log with option for long-term storage in cloud storage)
  • agent role memory (synthesized knowledge for specific roles like architect, coder, tester)
  • agent instance memory ((synthesized knowledge for specific instance of a roles like architect planning a specific project)
  • project memory (project-specific synthesized knowledge that cuts across agents)
  • cross-project memory (knowledge relevant for multiple projects)
  • user memory (user-specific knowledge)

Different stores for different kinds of memory. Multiple stores may be used for any type of memory. Agents will be able to access all relevant memory types. There will be dedicated tools for querying and updating the brain.

Stores:

  • in-memory
  • JSONL files (for session logs)
  • Markdown files (local or in the cloud)
  • Sqlite
  • Postgres with pgvector for RAG
  • Kubernetes custom resources (maybe for Claudenetes)

Problem: thinking about managing sensitive information

TBD: do we need forget APIs

Implement in Rust, because why not?

Agentic Issue Manager

Generic issue manager for agents with pluggable backends. Inspired by Steve Yegge's beads

Synergy with the Brain project is TBD. Seems like it should be part of the project memory and/or agent instance memory.

Operations include:

  • Create hierarchical issues (limited by backend only, e.g. Github Issues can do eight levels of hierarchy)
  • Labels
  • Drill down and search
  • Assign
  • Status (New, Assigned, In-progress, In-review, Done)
  • Search by any attribute

Backends as plugins:

  • GitHub issues
  • Linear
  • Kubernetes custom resources (for Claudenetes)
  • even beads :-)

Interface:

  • API
  • CLI
  • MCP

Implement in Rust, because why not?

Claudenetes

Agentic software development. Inspired (and horrified) by Steve Yegge's Gas Town

The idea is to run multiple AI agents that will automatically take over large projects and drive them to completion. The orchestration will be done by Kubernetes.

There will be persistent agent role and agent instance CRs (custom resources) that will maintain long term learning and task state

Agent instances will be assigned to issues and will be able to break them down to sub-issues and assign them to themselves, other agents or humans (or a dedicated issue manager will be responsible for assignment)

Kubernetes normal reconciliation loop will keep everything moving alog. Agents will watch relevant issue CRs, take action and update their CRs. Parent agents will watch their children's CRs and take proper action when all sub-issues are done or when an issue is in a bad state (failure or no update for a long time).

The entire state of the project is always reflected in the issue manager. Agent learnings are saved in the Brain.

Claudenetes can run locally on a KinD cluster or on a remote cluster. Target git repo/repos are cloned to node local storage, or in KinD just use mapping host path. Agents use git worktrees to work in parallel on the same git repo without stepping on each other's toes.

Other ideas: CoW - Council of Wells - Multiple models gang up and perform reviews of planning, architecture, implementation and tests and discuss between them to identify problems and suggest improvements.

See https://www.youtube.com/shorts/wvURinjb5Zk

TBD:

  • PR management (rebase, resolve conflicts, merge). is it the job of a dedicated agent or each agent is responsible? probably a sub-task
  • How to run Claude exactly? via Agent SDK or with -p ?
  • Do we even need Claude Code or can just hit LLMs directly with AI-6? it's probably best to use Claude Code.
  • Implement in Golang or Rust (or both)?

See prior art:

Demo projects

Auto Web Login

Fully-autonomous AI software engineer

  • Watch GitHub issues
  • Watch Slack channel
  • Respond to new issues assigned to it
  • Create local branch
  • Make changes + tests
  • Run tests
  • Open PR
  • Watch for PR comments
  • Make further changes based on PR comments
  • Merge PR when approved
  • Deploy and monitor
  • Rollback if something goes wrong

Meetings

Project Meeting 4-January-2026

Agenda

  • [√] Happy New Year 🎉
  • [√] Book update
  • Sibling projects
  • Claude Code Deep Dive (CCDD) blog series

Actions items

  • Gigi - continue CCDD (one blog per week?)
  • Gigi - start working on Brain, check out neon DB free offering for Postgres + pgvector.

Project Meeting 21-September-2025

Agenda

  • [√] Gigi - switch to uv (postpone until the book is done to avoid confusing the readers with two setups switching in the middle of the book)
  • Read the docs integration

Actions items

Project Meeting 7-September-2025

Agenda

  • Read the docs integration

Actions items

Project Meeting 31-August-2025

Agenda

Actions items

  • [√] Gigi - extend Github analyzer example to use sub-agents
  • [√] Gigi - get A2A into working shape
  • [√] Gigi - Add auth support for A2A
  • Saar - Generate docs for AI-6. Continue with Sphinx / ReadTheDocs

Project Meeting 11-August-2025

Agenda

Actions items

Project Meeting 2-August-2025

Agenda

  • Review status of AI-6
  • Book
  • Discuss next steps

Action Items

Project Meeting 10-may-2025

AI-6 Status

  • Gigi - go over the AI-generated code and make it right (test code in _send, warning in validate message, summary not implemented)
  • Gigi - Partially MCP in addition to our tools (can discover the tools)
  • Saar - Adding settings to Chainlit app to select model and disable tools.

Ideas

  • Saar is hitting rate limits, explore dynamic + automatic provider switching (rotate calls to different providers under the covers)
  • Explore local models - Qwen 2.5 is too slow.

Agenda

  • Discuss development process, branches, PRs, etc
  • Tooling - ruff √
  • CI/CD - Github actions to run linters, formatters, tests, etc on PRs
  • Async I/O ???

Action Items

  • Gigi - fix unit tests (some failing)

  • Gigi - run linters, formatters, etc on PRs

  • Gigi - Engine should reject command tool calls with pipes gracefully (later may be support pipes)

  • Gigi - Finalize MCP integration

  • Gigi - Github actions workflo to run tests on PRs and direct push to main

  • Gigi - Look into migrating the engine to Async IO

  • Gigi - try devestral on ollama

  • Saar - Use models from model_info.py

  • Saar - Look into usable local models

  • Saar - Add tool calls view to Chainlit UI

Project Meeting 3-may-2025

AI-6 Status

  • Gigi - Fixed tests and merge new tools
  • Gigi - Engine + Tool configuration
  • Gigi - Look into MCP

Action Items

  • Gigi - go over the AI-generated code and make it right (test code in _send, warning in validate message, summary not implemented)
  • [/] Gigi - Integrate MCP in addition to our tools, think about MCP tool discovery, local vs remote
  • Saar - develop dOpus with AI-6
  • Saar - show tool list in a chainlit side panel

Project Meeting 26-apr-2025

AI-6 Status

  • Multi-provider support (OpenAI + Ollama providers)
  • Local models (via ollama provider)
  • Memory (via session)
  • Streaming
  • New tools - sed, awk and patch

Working with OpenHands

OpenHands (previously OpenDevin) is an open source AI software engineer. Very easy to use and works well.

docker run -it --rm --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.33-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
    docker.all-hands.dev/all-hands-ai/openhands:0.33

Actions items

  • Gigi - Fix tests and merge new tools
  • Gigi - Engine + Tool configuration
  • Gigi - Look into MCP
  • Saar - Review current state
  • Saar - look into making bot output nicer on slack (e.g. render markdown as markdown)
  • Saar - Work on dOpus

Project Meeting 5-apr-2025

  • Discuss summarization when context window feels up.

  • More file system tools

    • sed (for string replace and other edits on a file)
    • patch

  • Support for Anthropic built-in text editor tool

  • Streaming (display content as it is generated)

  • Managing long-running processes (e.g. ollama serve)

  • Support for multiple LLM providers

    • OpenAI
    • Ollama + Gemma 3 function calling (no tool use on Ollama)
    • Anthropic
    • Gemini 2.5 pro experimental (free tier)
    • Llamma 4 Scout

Action Items

  • Saar/Gigi - Multiple LLM Providers
  • Saar - Research Ollama + Gemma 3 function calling
  • Saar - look into making bot output nicer on slack (e.g. render markdown as markdown)
  • Gigi - Look into MCP
  • Gigi - Engine + Tool configuration

Project Meeting 29-mar-2025

Action Items

  • Gigi - Check out chainlit
  • Gigi - Issues
    • AI-6 slack should ignore messages that mention other users
    • AI-6 slack bot should leave the channel on app exit
    • Bootstrap tool to restart itself (AI-6 runs new code after changes)

Project Meeting 22-mar-2025

Action Items

  • Gigi - Put this in the repo as planning.md
  • Gigi - Implement Slack UI
  • Gigi - Auto-discovery of tools
  • Saar - Local AI end to end

Items for discussion

  • MVP is done. should we do Github release or at least tag it? Give people (and us) some reference points to progress.
    • Just a tag!
  • MCP - Model Context Protocol
  • Multiple LLM providers
    • Engine receives multiple OpenAI instances + metadata on each one
    • AI-6 may swtich providers depending on task
    • User can always request specific provider
    • Default provider will be configured
  • Local AI models
    • Lamma 3.2 works, but not great
    • Need to experiment with others and find at least one good model
  • Tool configuration
    • Currently only env variables are available
    • Proposal
      • pass optional config file path via env variable
      • Each tool can have its own config file format and know how to parse it
      • All tool config files must be placed under a standard root directory
      • This way all tools can get permissions to read this directory only and not get access to arbitrary dirs
      • Subdirs based on tool hierarchy are recommended to avoid conflicts
      • If the config file is Python module the tool may import it dynamically (tool's business. AI-6 doesn't care)
      • It's possible to aggregate all config files into one big config file (one stop shop for users) and then some config helper will break it down into separate files expected by tools and update the environment.

Code walk

Project Meeting 16-mar-2025

Action Items

Current Status

  • AI-6
    • Works end to end
    • The engine implements the agentic loop
    • An extensible tool system exists with one implemented tool (ls)
    • CLI frontend works
    • Everything is implemented in Python
  • Slacker
    • Works

Items for discussions

  • Should we put this document in the ai-six repo? develop in the open? yes

  • What tools should we implement next?

  • Permissions

    • File access - root dir restrictions per tool (e.g. ls can be on everything, but rm only on some directories)
    • Prefer explicit whitelisting (default is no permissions)
    • Holistic approach - run ai6 as a specific user, it can't do more damage than its user permissions allow
    • Single user or perhaps optional users for specific tools.

  • Voice UI

  • Observability

    • tokens (input and output)
    • for Slack UI capture tool calls as messages on a thread
    • context window size?

  • Inception - AI-6 launching AI-6 as a tool (parent doesn't see context of child, just final output)

  • Non user input - Some server can listen to events and invoke AI-6 and based on the answer take some action

Kickoff Meeting 8-mar-2025

Action items this meeting

  • Pick a name (AI-6)
  • Schedule future meetings
  • Try "Code with Me"
  • Pick actions items for next meeting
  • Create #ai-6 slack channel

Pick a name

  • AI Whisperer
  • AI-6 (MI-6 and SEAL Team 6)
  • Xeno AI (sci-fi, non-human)
  • Agent Smith (Matrix)
  • AI Stein (Einstein)
  • Tool AI

ChatGPT was very impressed with AI-6! image

Architecture / Roadmap

  • the core engine is responsible for the agentic loop
  • the core engine is NOT responsible for tool saftey, security, etc
  • plugin-in architecture where arbitrary tools can be added
  • each plugin is resposible for interacting with the target tool
  • generic (extensible) config file for all the tools and/or config file per tool
  • external tool registry
  • The core engine can run locally or in the cloud
  • Support multiple LLM providers (local + managed)
  • engine plugin API (programming language native, REST, gRPC)
  • engine API - allow programmatic access to AI-6

Question: do we need an architecture document? Answer: No! later.

Self-hosted + managed LLM providers

  • Support multiple providers (OpenAI, Anthropic, Gemini, etc)
  • AI service access is easy, but can get expensive
  • Free-tier of managed AI providers may be sufficient for development
  • Local is ideal if performs well

Programming languages

  • Slack has support for Typescript, Python with supposedly good libraries
  • Engine - options: Rust, Go and Python and possibly even ALL of them
  • Tool plugins in engine language or any language (docker container, REST, gRPC)
  • Frontend - Typescript

Tools

  • Kubernetes via kubectl
  • Git
  • Github
  • Docker
  • Shell (inside a Docker container) - super risky. maybe not?
  • Any API (Youtube, Spotify, Facebook)
  • Homebrew
  • Tool discovery and request

Session persistence not needed. Tools can persist WIP themselves (e.g. in github PRs for a coding project)

UI

  • Slack
  • CLI
  • Web (can run on mobile too)

Github repo structure

  • Monorepo
  • Separate repo for each component
  • Hybrid
    • Monorepo for project-specific components
    • Separate repos for standalone libraries (e.g. slacker)
  • Github organization (never done it)

Decision: start with Github organization + monorepo. Move to hybrid if needed

MVP

  • Simple UI - Slack + CLI
  • Core (copy and extend k8s-ai)
  • Initial built-in Tools (k8s + git)
  • No plugins

Done!

Reference

  • k8s-ai - A Simple Python CLI PoC of a Kubernetes AI agent
  • slacker - Control Slack programmatically

Resources