AgentWatch

An open-source observatory for public reports of AI-agent incidents.

🔗 Live

🖥️ Dashboard	https://agentwatch-web.onrender.com
📡 API docs	https://agentwatch-api-7mhz.onrender.com/docs
📖 Documentation	https://kohsheen1234.github.io/Open-source-AI-Incident-Observatory/

The dashboard and API run on Render's free tier, which sleeps after ~15 min idle — the first request may take 30–60s to wake, then it's fast. The docs site is always on.

The dashboard, API docs, and Grafana metrics — running the full stack against live Hacker News incidents, classified by a local open-weight model.

As AI systems increasingly act on their own — running tools, taking actions, and operating with growing autonomy — people are posting about what happens when those systems behave in unexpected or unintended ways: an agent that deletes the wrong files, ignores an instruction, takes an action nobody approved, or behaves deceptively. These reports are scattered across forums and social platforms, and the original posts often disappear.

AgentWatch's goal is to turn that scattered, disappearing evidence into a durable, searchable, and analysable record — so researchers and safety teams can measure how often these incidents happen, what kinds occur, and how the picture changes over time.

🔭 Project status

The end-to-end pipeline works today: AgentWatch collects public reports from pluggable sources, preserves each as tamper-evident hashed evidence, normalises them into de-duplicated incidents, classifies each with a pluggable abstain-capable classifier (measured by a labelled evaluation set and a prompt-regression gate), exposes everything through a documented, authenticated HTTP API and a review dashboard, ships Prometheus metrics + a provisioned Grafana dashboard, and comes up as a whole with a single docker compose up behind a Caddy reverse proxy (auto-HTTPS in production). Everything described here is runnable now — one command brings up the full stack.

---

Live demo & deployment

• Documentation site (live): https://kohsheen1234.github.io/Open-source-AI-Incident-Observatory/ — published automatically from main via GitHub Actions.

• Run the interactive app yourself: one make up locally (below), or one-click to the cloud:

  

  The render.yaml blueprint provisions the API, the dashboard, and a managed Postgres. (A live interactive app needs a host account; the docs site above is always-on and needs nothing.)

Screenshots

Overview	How it works (methodology)

Incident Explorer	Review Queue

API — OpenAPI docs	Grafana — metrics

Architecture

Five stages — collect → preserve → normalise → classify → serve — with evaluate and review as quality loops around the classifier. PostgreSQL is the single source of truth; a FastAPI service is the only access layer; the React dashboard and any external consumer use that same API.

📐 Read the full system design — every decision, why it was made, and its tradeoffs.

Scale: designed for it, validated locally

This is a portfolio system — the aim is to show it's designed and validated for scale, not to pay to operate at scale. A repeatable local load test (agentwatch bench --count 100000) runs 100k synthetic incidents through the real pipeline on SQLite and reports:

• ingestion ~4,100 records/sec, with 30% duplicates correctly suppressed
• classification ~8,600 incidents/sec (baseline provider)
• GET /incidents p50 ≈ 188 ms at 70k rows — the identified bottleneck (a per-request COUNT over a latest-classification join; the fix is keyset pagination + a materialised latest-classification column + Postgres indexes)

Deliberate-failure tests (tests/test_failure_modes.py) cover collector timeouts (isolated), mid-batch storage failure (atomic rollback), duplicate ingestion (idempotent), malformed input (abstain), and bad model output (abstain). Full write-up: Benchmarks & failure modes.

Demonstrated (built & tested)	Designed for, not yet required
Single-node ingestion (~4.1k rec/s)	Horizontal ingestion workers
Content-hash deduplication	Distributed deduplication
Pluggable storage (StorageBackend → LocalArtifactStore)	S3ArtifactStore (interface in place)
Pluggable sources (DataSource) + scheduled/CLI collection	Durable job queue
Pluggable classifier (baseline / Ollama / Anthropic)	Larger hosted models at volume
Prometheus metrics + Grafana	Multi-region monitoring
Portable Postgres / SQLite	Read replicas / sharding

Run the whole system

With Docker installed:

make up      # builds and starts: db, api, dashboard, prometheus, grafana, caddy

Then open:

• http://localhost:8080 — the review dashboard
• http://localhost:8080/api/docs — the API's OpenAPI docs
• http://localhost:8080/grafana — the Grafana metrics dashboard

Populate it with sample data (runs inside the stack):

docker compose -f deploy/docker-compose.yml exec api \
  sh -c "agentwatch collect --source replay && agentwatch classify --provider baseline"

Stop it with make down. See docs/deployment.md for the VPS + HTTPS guide. Prefer to run pieces directly on your machine? See Quickstart.

What's built today

AgentWatch is being built in layers. The foundation layer — the part that is complete and tested — provides:

Capability	What it does
Typed configuration	All settings (database, storage location, log level, secrets) load from environment variables with safe defaults, validated at startup.
Structured logging	JSON logs via structlog, ready for aggregation and machine parsing.
Database schema	A five-table relational model (below) that captures raw evidence, normalised incidents, machine classifications, human reviews, and collection runs.
Migrations	Versioned schema changes via Alembic; the same schema runs on PostgreSQL (production) and SQLite (fast tests).
Containerised database	A one-command PostgreSQL service via Docker Compose.
Pluggable collectors	A DataSource interface with three adapters: Hacker News (live), Replay (bundled fixtures, no credentials needed), and Reddit (opt-in). Adding a source is one new file.
Tamper-evident evidence	Every collected item is stored verbatim on disk, named by its SHA-256 hash, so evidence survives deletion of the original.
Normalise + de-duplicate	Collected items become de-duplicated incidents; re-collecting the same content adds nothing. Author identifiers are hashed.
Reliable collection	A CLI and optional scheduler run collection with retries, per-source failure isolation, and a recorded run history.
Pluggable classifier	An LLMProvider interface with three backends: a deterministic Baseline (default, no dependencies), Ollama (local open-weight models), and optional Anthropic. Structured JSON output is validated; malformed output is retried, then abstained.
Abstain-capable taxonomy	Ten incident types plus an explicit insufficient_evidence outcome, so the system distinguishes "no incident" from "not enough evidence".
Measured quality	A frozen 131-example labelled set (full taxonomy + keyword-misleading hard negatives) scored across a majority → keyword → local-model baseline ladder: macro-F1 on both dimensions, per-class precision/recall, selective accuracy at coverage, abstention precision/recall, calibration, and failure cases — guarded by a regression test.
Documented HTTP API	A FastAPI service (list / filter / detail / review / stats / CSV export) with auto-generated OpenAPI docs and optional API-key auth on writes.
Review dashboard	A React single-page app (overview, incident explorer, review queue) that consumes the API — reviewers accept, override, or flag classifications.
Metrics & dashboards	The API exposes Prometheus metrics; Prometheus scrapes them and a provisioned Grafana dashboard visualises incidents, classifications, abstention rate, and collection-run health.
One-command stack	docker compose up brings up db, API, dashboard, Prometheus, Grafana, and a Caddy reverse proxy (auto-HTTPS in production) on one network.
Test suite	Every component is covered by tests that run in under a second.

If you clone this repository, all of the above runs and passes. Nothing here is a placeholder.

The data model

The schema is the heart of the foundation. It is designed around a simple idea: keep the original evidence separate from any interpretation of it, and record who interpreted it and how.

collection_runs        — one row per collection job (when it ran, what it found, any error)
        │
        ▼
raw_artifacts          — the untouched source record + a SHA-256 content hash
        │                 (evidence is preserved even if the original is deleted)
        ▼
incidents              — the normalised, de-duplicated report (title, body, source, date)
        │                 author identifiers are stored HASHED, never in the raw
        ▼
classifications        — a machine label for an incident (type, severity, confidence…)
        │                 records the model and prompt version used
        ▼
reviews                — a human decision on a classification (accept / override / reject)

Two design choices worth calling out, because they shape everything downstream:

• Raw evidence is immutable and hashed. Every source record is stored verbatim with a SHA-256 hash, so the evidence survives even if the original post is removed, and any later tampering is detectable.
• Author privacy by default. Author identifiers are stored as salted hashes, never in plaintext. The schema has no column for a raw author name.

See docs/data-model.md for the full table-by-table reference.

Quickstart

Requirements: Python 3.12+ and Docker.

# 1. Install the package and dev tools (a virtualenv is recommended)
pip install -e ".[dev]"

# 2. Start PostgreSQL (runs on host port 5433 to avoid clashing with a local Postgres)
make db-up

# 3. Point AgentWatch at it and create the schema
cp .env.example .env
make migrate

# 4. Run the test suite
make test

You now have a running database with the full AgentWatch schema, verified by tests.

To run against SQLite instead (no Docker needed), leave AGENTWATCH_DATABASE_URL unset — it defaults to a local SQLite file, which is exactly what the test suite uses.

Collecting incidents

Once the schema exists, collect incidents with the agentwatch CLI. The replay source needs no credentials and works immediately, so you can see the full pipeline end to end:

# Collect from the bundled replay fixtures (no credentials required)
agentwatch collect --source replay

# Collect live from Hacker News, or from every configured source
agentwatch collect --source hackernews
agentwatch collect --source all --since-hours 168

# Run collection continuously on a schedule
agentwatch schedule --interval-minutes 60

Each run stores the original evidence under AGENTWATCH_ARTIFACT_DIR (as <source>/<year>/<month>/<sha256>.json), writes de-duplicated incidents to the database, and records a row in collection_runs. Re-running over the same window adds nothing new.

Sources:

• replay — bundled sample incidents; the credential-free default.
• hackernews — live via the public Hacker News (Algolia) API; no key required.
• reddit — opt-in; set AGENTWATCH_REDDIT_CLIENT_ID and AGENTWATCH_REDDIT_CLIENT_SECRET and install the extra (pip install -e ".[reddit]").

Classifying incidents

Once incidents exist, classify the ones that have no classification yet:

# Deterministic baseline classifier (default; no model server or network needed)
agentwatch classify --provider baseline

# Classify with a local open-weight model served by Ollama
agentwatch classify --provider ollama

Each classification records the incident type, severity, confidence, whether the model abstained, and the exact model_name and prompt_version used — so results are reproducible and auditable.

Measuring classifier quality

The classifier is scored against a frozen, deliberately hard 131-example labelled set (full taxonomy, 24 keyword-misleading hard negatives, 14 under-evidenced cases) across a ladder of baselines:

agentwatch eval --provider majority    # constant-class floor
agentwatch eval --provider baseline    # deterministic keyword classifier (default)
agentwatch eval --provider ollama      # a real local model on identical data

Each run reports macro-F1 on both the relevance and incident-type dimensions, per-class precision/recall, selective accuracy at a coverage level, abstention precision/recall, calibration, cost, latency, and the ten most-confident failure cases. Measured on the frozen set:

System	Incident macro-F1	Relevance macro-F1	Selective acc @ coverage	Cost	Latency
Majority (floor)	0.025	0.277	0.09 @ 1.00	$0	~0 ms
Keyword baseline	0.273	0.189	0.37 @ 0.29	$0	~0 ms
Local qwen2.5:7b	0.749	0.747	0.72 @ 0.94	$0	~4 s

The local model nearly triples the baseline's discriminative score and handles the keyword-misleading hard negatives the baseline can't (not_relevant F1 0.00 → 0.69). The evaluation also caught a real bug: an earlier 0.00 on not_relevant turned out to be a schema that discarded the model's correct "not an incident" verdicts — the kind of thing only per-class metrics expose. Full analysis and honest limitations in docs/evaluation.md.

A test (tests/test_eval.py) runs this evaluation and fails if macro-F1 drops below a committed floor or the baseline stops beating the majority floor, so a prompt or model change that regresses quality is caught in CI.

See docs/evaluation.md for the dataset design, annotation methodology, full results, and honest limitations.

The HTTP API

Serve the API (auto-generated OpenAPI docs at /docs):

agentwatch serve --host 127.0.0.1 --port 8000

Method & path	Purpose
GET /health	Liveness check
GET /incidents	List incidents (filter by source, incident_type, abstained, min_severity; paginated with limit/offset)
GET /incidents/{id}	Incident detail with all classifications and reviews
POST /incidents/{id}/review	Record a human review (accept / override / false_positive)
GET /stats	Summary counts and abstention rate
GET /exports/incidents.csv	Export incidents as CSV

Reads are public. If AGENTWATCH_API_KEY is set, writes (review) and CSV export require an X-API-Key header — so a reviewer can run it locally with zero config, while production can lock it down.

The web dashboard

The dashboard is a React single-page app (Vite + TypeScript + Tailwind, charts with Recharts) in frontend/. It only ever talks to the HTTP API. Run it locally:

cd frontend
npm install
VITE_API_URL=http://localhost:8000 npm run dev   # dev server on http://localhost:5173

The whole stack (API + web behind Caddy) also comes up with make up — see Run the whole system.

Pages: Overview (mission, pipeline, KPIs, and interactive charts), Incident Explorer (filterable table with colored type/severity badges and per-incident evidence), and Review Queue (accept / override / flag a classification).

Configuration

All configuration is read from environment variables prefixed AGENTWATCH_ (see .env.example):

Variable	Default	Purpose
AGENTWATCH_DATABASE_URL	local SQLite file	SQLAlchemy database URL
AGENTWATCH_ARTIFACT_DIR	./artifacts	Where raw evidence files are stored
AGENTWATCH_AUTHOR_HASH_SALT	change-me-in-production	Salt for hashing author identifiers
AGENTWATCH_LOG_LEVEL	INFO	Log verbosity
AGENTWATCH_ENVIRONMENT	local	Deployment environment label
AGENTWATCH_REDDIT_CLIENT_ID	(unset)	Enables the opt-in Reddit source
AGENTWATCH_REDDIT_CLIENT_SECRET	(unset)	Enables the opt-in Reddit source
AGENTWATCH_API_KEY	(unset)	If set, required (X-API-Key) for API writes and export

The web app is configured at build time with VITE_API_URL (the API base URL it calls).

Tech stack

Python 3.12 · SQLAlchemy 2.0 · Alembic · Pydantic · FastAPI · uvicorn · React + Vite + TypeScript + Tailwind + Recharts (frontend) · httpx · tenacity · APScheduler · structlog · PostgreSQL 16 · Docker Compose · pytest · ruff.

Documentation

Longer-form docs live in docs/ and are published as a MkDocs site:

• Overview — what AgentWatch is and how the foundation fits together
• Architecture — the components that exist today
• Data model — full schema reference
• Development — setup, testing, and how to add a migration

To preview the docs site locally:

pip install -e ".[docs]"
mkdocs serve

Continuous integration

Every push and pull request runs GitHub Actions: ruff linting and the full pytest suite — which includes the classifier evaluation regression gate, so a change that drops macro-F1 below the committed floor fails CI. A separate docs workflow builds the MkDocs site with --strict and publishes it to GitHub Pages on every push to main. A keep-alive workflow pings the live demo every ~10 minutes so the free-tier services stay awake (and can be triggered manually from the Actions tab).

Project layout

agentwatch/        # the package
  config.py        # typed settings from the environment
  logging.py       # structured JSON logging
  hashing.py       # content + author hashing
  collectors/      # DataSource protocol + adapters (hackernews, replay, reddit)
  storage/         # tamper-evident artifact file store
  pipeline/        # ingest (persist/normalise/dedupe) + collection orchestration
  classify/        # taxonomy, prompt, providers (baseline/ollama/anthropic), classifier
  eval/            # labelled dataset, metrics, evaluation runner
  api/             # FastAPI app, schemas, queries, auth
  sources.py       # source + provider registry / defaults
  cli.py           # `agentwatch` command-line interface
  scheduler.py     # APScheduler-based recurring collection
  db/              # SQLAlchemy models, base, portable types, session management
frontend/          # React + Vite + TypeScript + Tailwind single-page app (the web UI)
migrations/        # Alembic migration environment and versions
deploy/            # docker-compose service definitions
docs/              # MkDocs documentation site
tests/             # test suite

Contributing

See CONTRIBUTING.md.

Licence

See LICENSE.