development-setup.md

Local Development Setup

This guide walks you through setting up Open Notebook for local development. Follow these steps to get the full stack running on your machine.

Prerequisites

Before you start, ensure you have the following installed:

• Python 3.11+ - Check with: python --version
• uv (recommended) or pip - Install from: https://github.com/astral-sh/uv
• SurrealDB - Via Docker or binary (see below)
• Docker (optional) - For containerized database
• Node.js 18+ (optional) - For frontend development
• Git - For version control

Step 1: Clone and Initial Setup

# Clone the repository
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook

# Add upstream remote for keeping your fork updated
git remote add upstream https://github.com/lfnovo/open-notebook.git

Step 2: Install Python Dependencies

# Using uv (recommended)
uv sync

# Or using pip
pip install -e .

Step 3: Environment Variables

Create a .env file in the project root with your configuration:

# Copy from example
cp .env.example .env

Edit .env with your settings:

# Database
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development

# Credential encryption (required for storing API keys)
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-dev-secret-key

# Application
OPEN_NOTEBOOK_PASSWORD=  # Optional password protection
DEBUG=true
LOG_LEVEL=DEBUG

AI Provider Configuration

After starting the API and frontend, configure your AI provider via the Settings UI:

1. Open http://localhost:3000 → Settings → Models
2. Click Add Credential → Select your provider
3. Enter your API key (get from provider dashboard)
4. Click Save, then Test Connection
5. Click Discover Models → Register Models

Popular providers:

• OpenAI - https://platform.openai.com/api-keys
• Anthropic (Claude) - https://console.anthropic.com/
• Google - https://ai.google.dev/
• Groq - https://console.groq.com/

For local development, you can also use:

• Ollama - Run locally without API keys (see "Local Ollama" below)

Note: API key environment variables (e.g., OPENAI_API_KEY) are deprecated. Use the Settings UI to manage credentials instead.

Step 4: Start SurrealDB

Option A: Using Docker (Recommended)

# Start SurrealDB in memory (publish the port on localhost only — the
# database uses default credentials, so never publish it on 0.0.0.0)
docker run -d --name surrealdb -p 127.0.0.1:8000:8000 \
  surrealdb/surrealdb:v2 start \
  --user root --pass password \
  memory

# Or with persistent storage
docker run -d --name surrealdb -p 127.0.0.1:8000:8000 \
  -v surrealdb_data:/data \
  surrealdb/surrealdb:v2 start \
  --user root --pass password \
  file:/data/surreal.db

Option B: Using Make

make database

Option C: Using Docker Compose

docker compose up -d surrealdb

Verify SurrealDB is Running

# Should show server information
curl http://localhost:8000/

Step 5: Run Database Migrations

Database migrations run automatically when you start the API. The first startup will apply any pending migrations.

To verify migrations manually:

# API will run migrations on startup
uv run python -m api.main

Check the logs - you should see messages like:

Running migration 001_initial_schema
Running migration 002_add_vectors
...
Migrations completed successfully

Step 6: Start the API Server

In a new terminal window:

# Terminal 2: Start API (port 5055)
uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055

# Or using the shortcut
make api

You should see:

INFO:     Application startup complete
INFO:     Uvicorn running on http://0.0.0.0:5055

Verify API is Running

# Check health endpoint
curl http://localhost:5055/health

# View API documentation
open http://localhost:5055/docs

Step 7: Start the Frontend (Optional)

If you want to work on the frontend, start Next.js in another terminal:

# Terminal 3: Start Next.js frontend (port 3000)
cd frontend
npm install  # First time only
npm run dev

You should see:

> next dev
  ▲ Next.js 16.x
  - Local:        http://localhost:3000

Access the Frontend

Open your browser to: http://localhost:3000

Verification Checklist

After setup, verify everything is working:

• SurrealDB: curl http://localhost:8000/ returns content
• API: curl http://localhost:5055/health returns {"status": "ok"}
• API Docs: open http://localhost:5055/docs works
• Database: API logs show migrations completing
• Frontend (optional): http://localhost:3000 loads

Development Workflows: When to Use What?

Workflow	Use Case	Speed	Production Parity
Local Services (make start-all)	Day-to-day development, fastest iteration	⚡⚡⚡ Fast	Medium
Docker Compose (make dev)	Testing containerized setup	⚡⚡ Medium	High
Local Docker Build (make docker-build-local)	Testing Dockerfile changes	⚡ Slow	Very High
Multi-platform Build (make docker-push)	Publishing releases (see Release Process)	🐌 Very Slow	Exact

Local services give hot reload, direct log access and easy debugging; Docker Compose (examples/docker-compose-dev.yml via make dev, examples/docker-compose-full-local.yml via make full) is closer to production. Use make docker-build-local before touching anything Docker-related in a PR.

Starting Services Together

Quick Start All Services

make start-all    # SurrealDB + API + worker + frontend
make status       # see what's running
make stop-all     # stop everything

Individual Terminals (Recommended for Development)

Terminal 1 - Database:

make database

Terminal 2 - API:

make api

Terminal 3 - Background worker (required for podcasts, embeddings, source processing):

make worker-start

Terminal 4 - Frontend:

cd frontend && npm run dev

Performance Tips

1. Use make start-all instead of Docker for daily work
2. Keep SurrealDB running between sessions (make database)
3. Use make docker-build-local only when testing Dockerfile changes
4. Skip multi-platform builds until ready to publish
5. Clean caches when things get weird: make clean-cache, docker system prune -a

Development Tools Setup

Pre-commit Hooks (Optional but Recommended)

Pre-commit hooks run configured checks automatically before each commit, mirroring the CI gates so local commits fail for the same reasons PRs would. The config at .pre-commit-config.yaml wires up:

Tool	What it checks	CI equivalent
ruff (lint)	Python lint rules (E, F, I)	ruff check .
ruff (format)	Python formatting (line-length 88)	Not yet gated
mypy	Python type correctness	python -m mypy .
pre-commit-hooks	Large files, merge conflicts, YAML/TOML syntax, trailing whitespace, EOF newlines	—

Pre-commit is already included in the project's dev dependencies. Install the hooks and they'll run on every git commit:

uv run pre-commit install

Running manually:

# Check all files (useful after changing hook config)
uv run pre-commit run --all-files

# Run a specific hook only
uv run pre-commit run ruff --all-files

Skipping hooks temporarily:

# Skip all hooks for a single commit
git commit --no-verify

# Skip a specific hook (e.g. slow mypy run)
SKIP=mypy git commit

Updating hook versions:

uv run pre-commit autoupdate

Keep the rev: pins in .pre-commit-config.yaml in sync with the versions listed in pyproject.toml under [dependency-groups] dev.

Code Quality Commands

# Lint Python code (auto-fix)
make ruff
# or: ruff check . --fix

# Type check Python code
make lint
# or: uv run python -m mypy .

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=open_notebook

Common Development Tasks

Running Tests

# Run all tests
uv run pytest

# Run specific test file
uv run pytest tests/test_notebooks.py

# Run with coverage report
uv run pytest --cov=open_notebook --cov-report=html

Creating a Feature Branch

# Create and switch to new branch
git checkout -b feature/my-feature

# Make changes, then commit
git add .
git commit -m "feat: add my feature"

# Push to your fork
git push origin feature/my-feature

Updating from Upstream

# Fetch latest changes
git fetch upstream

# Rebase your branch
git rebase upstream/main

# Push updated branch
git push origin feature/my-feature -f

Troubleshooting

"Connection refused" on SurrealDB

Problem: API can't connect to SurrealDB

Solutions:

1. Check if SurrealDB is running: docker ps | grep surrealdb
2. Verify URL in .env: Should be ws://localhost:8000/rpc
3. Restart SurrealDB: docker stop surrealdb && docker rm surrealdb
4. Then restart with: docker run -d --name surrealdb -p 127.0.0.1:8000:8000 surrealdb/surrealdb:v2 start --user root --pass password memory

"Address already in use"

Problem: Port 5055 or 3000 is already in use

Solutions:

# Find process using port
lsof -i :5055  # Check port 5055

# Kill process (macOS/Linux)
kill -9 <PID>

# Or use different port
uvicorn api.main:app --port 5056

Module not found errors

Problem: Import errors when running API

Solutions:

# Reinstall dependencies
uv sync

# Or with pip
pip install -e .

Database migration failures

Problem: API fails to start with migration errors

Solutions:

1. Check SurrealDB is running: curl http://localhost:8000/
2. Check credentials in .env match your SurrealDB setup
3. Check logs for specific migration error: make api 2>&1 | grep -i migration
4. Verify database exists: Check SurrealDB console at http://localhost:8000/

Migrations not applying

Problem: Database schema seems outdated

Solutions:

1. Restart API - migrations run on startup: make api
2. Check logs show "Migrations completed successfully"
3. Verify /migrations/ folder exists and has files
4. Check SurrealDB is writable and not in read-only mode

Optional: Local Ollama Setup

For testing with local AI models:

# Install Ollama from https://ollama.ai

# Pull a model (e.g., Mistral 7B)
ollama pull mistral

Then configure via the Settings UI:

1. Go to Settings → Models → Add Credential → Ollama
2. Enter base URL: http://localhost:11434
3. Click Save, then Test Connection
4. Click Discover Models → Register Models

Optional: Docker Development Environment

Run entire stack in Docker:

# Start all services
docker compose --profile multi up

# Logs
docker compose logs -f

# Stop services
docker compose down

Next Steps

After setup is complete:

1. Read the Contributing Guide - contributing.md
2. Explore the Architecture - Check the documentation
3. Find an Issue - Look for "good first issue" on GitHub
4. Set Up Pre-commit - Install git hooks for code quality
5. Join Discord - https://discord.gg/37XJPXfz2w

Getting Help

If you get stuck:

• Discord: Join our server for real-time help
• GitHub Issues: Check existing issues for similar problems
• GitHub Discussions: Ask questions in discussions
• Documentation: See code-standards.md and testing.md

---

Ready to contribute? Go to contributing.md for the contribution workflow.