Rewrite API into route modules, harden sessions, drop scraper + MCP

Lands the reorganisation documented in PROGRESS.md:

- Structure: api/src/index.ts becomes a slim composition root; every
  endpoint moves to api/src/routes/* (one module per domain) with
  shared middleware.ts and helpers.ts
- Security: server-side sessions (SHA-256 token hashes, real
  revocation on logout/password change), SESSION_SECRET removed,
  hono/csrf origin checks, secure headers (+ CSP in prod),
  request-size limits, proxy-aware login rate limiting, audited
  logins, /api/settings admin-only both directions
- Efficiency: bulk-loaded document items/payments, explicit column
  lists so signed-doc BLOBs stay out of list queries
- Removed: scraper/ workspace and /api/scraper/*, stale mcp/
  workspace, SERVICE_TOKEN, docs/INTEGRATIONS.md
- isarwebsites: fresh installs seed an 11-item Leistungskatalog and
  default business name; prompts/branding/docs updated

125 API tests green, api typecheck clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: lucas19919

.claude/launch.json

@@ -10,14 +10,28 @@
     {
       "name": "openleads-api",
       "runtimeExecutable": "npm",
-      "runtimeArgs": ["--prefix", "D:/Repos/openleads/api", "run", "dev"],
+      "runtimeArgs": ["--prefix", "D:/Repos/openleads/api", "run", "start"],
       "port": 8787
     },
     {
       "name": "openleads-web",
       "runtimeExecutable": "npm",
       "runtimeArgs": ["--prefix", "D:/Repos/openleads/web", "run", "dev"],
       "port": 5173
+    },
+    {
+      "name": "openleads-web-alt",
+      "runtimeExecutable": "npm",
+      "runtimeArgs": ["--prefix", "D:/Repos/openleads/web", "run", "dev", "--", "--port", "5175"],
+      "port": 5175,
+      "autoPort": false
+    },
+    {
+      "name": "openleads-web-redesign",
+      "runtimeExecutable": "npm",
+      "runtimeArgs": ["--prefix", "D:/Repos/openleads/web", "run", "dev", "--", "--port", "5176", "--strictPort"],
+      "port": 5176,
+      "autoPort": false
     }
   ]
 }

.claude/skills/setup-openleads/SKILL.md

@@ -1,31 +1,31 @@
 ---
 name: setup-openleads
 description: >-
-  Sets up a working OpenLeads instance from a fresh clone — installs the api/,
-  web/ and (optional) scraper/ workspaces, creates the .env files with freshly
-  generated secrets (SESSION_SECRET, SETTINGS_KEY, SERVICE_TOKEN), seeds a login,
-  optionally wires up local Ollama AI, then starts and verifies the dev servers.
-  Use this whenever someone wants to set up, install, bootstrap, configure, get
-  started with, onboard onto, or "run OpenLeads for the first time" — including
-  vague asks like "how do I run this?", "get this working", "set it up", or
-  "I just cloned this, now what?" while in the OpenLeads repo. Also the entry
-  point for production deployment, which it routes to deploy/DEPLOY.md.
+  Sets up a working OpenLeads instance from a fresh clone — installs the api/
+  and web/ workspaces, creates api/.env with a freshly generated SETTINGS_KEY,
+  seeds a login, optionally wires up local Ollama AI, then starts and verifies
+  the dev servers. Use this whenever someone wants to set up, install,
+  bootstrap, configure, get started with, onboard onto, or "run OpenLeads for
+  the first time" — including vague asks like "how do I run this?", "get this
+  working", "set it up", or "I just cloned this, now what?" while in the
+  OpenLeads repo. Also the entry point for production deployment, which it
+  routes to deploy/DEPLOY.md.
 ---
 
 # Set up OpenLeads
 
 Drive a first-time OpenLeads setup end to end: dependencies → secrets → login →
-running, verified servers. OpenLeads is a three-workspace monorepo — `api/`
-(Hono + `node:sqlite`), `web/` (React + Vite), `scraper/` (Anthropic SDK) — so
-"setup" means getting the pieces installed, configured, and talking to each
-other, not just one `npm install`.
+running, verified servers. OpenLeads is a two-workspace monorepo — `api/`
+(Hono + `node:sqlite`) and `web/` (React + Vite) — so "setup" means getting the
+pieces installed, configured, and talking to each other, not just one
+`npm install`.
 
 The canonical instructions live in the repo and may drift ahead of this skill.
 **Read these first and treat them as the source of truth** if they disagree with
 anything below:
 
 - `README.md` → "Quick start (development)" and "Configuration"
-- `api/.env.example` and `scraper/.env.example` → the authoritative env var list
+- `api/.env.example` → the authoritative env var list
 - `deploy/DEPLOY.md` → production (Docker Compose + nginx)
 
 ## Step 0 — Decide scope and gather inputs
@@ -38,10 +38,7 @@ Ask the user only what you can't infer, then proceed. Don't over-interview.
 2. **Login credentials** — you need a username and password to seed the first
    user. If they don't offer one, suggest `admin` and generate a strong password
    (and show it to them so they can save it).
-3. **Scraper?** Optional. Only needed to auto-discover leads, and it needs an
-   `ANTHROPIC_API_KEY`. Skip it if they don't have a key or don't ask — the CRM
-   and invoicing run fine without it.
-4. **Local AI (chat copilot)?** Optional. The Chat module needs an
+3. **Local AI (chat copilot)?** Optional. The Chat module needs an
    OpenAI-compatible endpoint; the default targets a **local Ollama**. Skip if
    they don't want it — the rest of the app still runs, only Chat is inert.
 
@@ -56,7 +53,7 @@ OpenLeads uses Node's built-in SQLite, which needs a recent Node:
   `node --version` and stop here with a clear message if it's older — nothing
   downstream will work.
 - `npm` ships with Node.
-- Ollama is only needed if they chose local AI (Step 5).
+- Ollama is only needed if they chose local AI (Step 4).
 
 ## Step 2 — API (`api/`)
 
@@ -72,22 +69,18 @@ already there, read it and only fill missing secrets; otherwise start from the
 example:
 
 1. Read `api/.env.example` to get the current, authoritative set of variables.
-2. Generate three independent secrets (see
-   [Generating secrets](#generating-secrets)):
-   - `SESSION_SECRET` — signs session cookies
+2. Generate one secret (see [Generating secrets](#generating-secrets)):
    - `SETTINGS_KEY` — AES-256-GCM key for credentials saved via the Settings UI
-   - `SERVICE_TOKEN` — bearer token the scraper uses to POST leads
-3. Write `api/.env` from the example with those three placeholders replaced.
-   Leave `NODE_ENV=development`, `DB_PATH`, ports, and the AI/SMTP block at their
+3. Write `api/.env` from the example with that placeholder replaced. Leave
+   `NODE_ENV=development`, `DB_PATH`, ports, and the AI/SMTP block at their
    example defaults unless the user asked otherwise.
 
-> **Why generate all three even in dev?** In development the app falls back to
-> insecure built-in defaults (with a warning) if these are unset, so it *would*
-> boot without them. But it **fails closed in production** — `api/src/auth.ts`
-> throws if `SESSION_SECRET` is the dev default under `NODE_ENV=production`, and
-> `api/src/secrets.ts` refuses to encrypt credentials without a real
-> `SETTINGS_KEY`. Generating them now means the same `.env` works when they later
-> flip to production, and it's the correct habit to model.
+> **Why generate it even in dev?** In development the app falls back to an
+> insecure built-in default (with a warning), so it *would* boot without it. But
+> it **fails closed in production** — `api/src/secrets.ts` refuses to encrypt
+> credentials without a real `SETTINGS_KEY`. Generating it now means the same
+> `.env` works when they later flip to production. (Sessions are stored
+> server-side in the DB — there is no session secret anymore.)
 
 **Seed the login** (creates the first user; the app has no signup):
 
@@ -110,50 +103,23 @@ No env file needed in dev — Vite serves on **http://localhost:5173** and proxi
 `web/vite.config.ts`). (If `README.md` quotes a different port, trust the actual
 `vite.config.ts`.)
 
-## Step 4 — Scraper (`scraper/`, optional)
-
-Skip unless the user wants automated lead discovery and has an Anthropic key.
-
-```
-cd ../scraper
-npm install
-```
-
-Create `scraper/.env` from `scraper/.env.example`, then set:
-
-- `ANTHROPIC_API_KEY` — their key from https://console.anthropic.com
-- `CRM_API_URL` — `http://127.0.0.1:8787` for local
-- `CRM_SERVICE_TOKEN` — **must exactly equal** the `SERVICE_TOKEN` you put in
-  `api/.env`. This is the single most common setup mistake: if they differ, the
-  API rejects every posted lead with 401 and nothing shows up. Copy the value
-  across literally.
-
-Verify offline first — this uses fixtures, no Claude calls, no network:
-
-```
-npm run dry-run
-```
-
-A live run is `npm start` (costs Anthropic tokens) — only run it if asked.
-
-## Step 5 — Local AI / Ollama (optional)
+## Step 4 — Local AI / Ollama (optional)
 
 Only if the user wants the Chat copilot. The API defaults
 (`AI_BASE_URL=http://localhost:11434/v1`, `AI_MODEL=llama3.1:8b`) target a local
 Ollama, so no `api/.env` change is needed — just make the models available:
 
 1. Install Ollama (https://ollama.com) if `ollama --version` fails.
-2. Pull the chat + embedding models named in `api/.env.example`:
+2. Pull the chat model named in `api/.env.example`:
    ```
    ollama pull llama3.1:8b
-   ollama pull nomic-embed-text
    ```
 
 If they'd rather use a hosted OpenAI-compatible endpoint, set `AI_BASE_URL`,
 `AI_MODEL`, and `AI_API_KEY` in `api/.env` instead (or configure it later in the
 Settings page, which encrypts the key at rest with `SETTINGS_KEY`).
 
-## Step 6 — Start and verify
+## Step 5 — Start and verify
 
 Start both dev servers (run them in the background / separate processes so they
 keep running):
@@ -173,24 +139,25 @@ Then **verify rather than assuming** — don't just report "done":
 - Confirm Vite is serving on 5173.
 - Tell the user to open **http://localhost:5173** and log in with the credentials
   from Step 2. The UI is in German (it targets the DACH market) — mention that so
-  they're not surprised.
+  they're not surprised. A fresh database starts with the isarwebsites
+  Leistungskatalog (website packages, hosting/Pflege, SEO) prefilled.
 
 Report exactly what came up and what's still optional/disabled (e.g. "Chat is
-inactive until you finish Step 5", "lead discovery off — no scraper key").
+inactive until you finish Step 4").
 
 ## Production
 
-Don't hand-roll this — OpenLeads ships one Docker image (web + API + scraper)
-and `deploy/DEPLOY.md` is the maintained walkthrough (Docker Compose, nginx +
-TLS, secrets via `api.env`/`scraper.env`, seeding the login in-container,
-scheduling the scraper via cron, backups). Read it and follow it, adapting the
-domain and secrets to the user's host. The same secret rules apply, plus
-`WEB_ORIGIN` must be the public origin and `NODE_ENV=production` (which makes the
-secret checks fail-closed — so the secrets are now mandatory, not optional).
+Don't hand-roll this — OpenLeads ships one Docker image (web + API) and
+`deploy/DEPLOY.md` is the maintained walkthrough (Docker Compose, nginx + TLS,
+secrets via `api.env`, seeding the login in-container, backups). Read it and
+follow it, adapting the domain and secrets to the user's host. The same secret
+rules apply, plus `WEB_ORIGIN` must be the public origin and
+`NODE_ENV=production` (which makes the secret checks fail-closed — so
+`SETTINGS_KEY` is then mandatory, not optional).
 
 ## Generating secrets
 
-Each secret is an independent long random string. Cross-platform:
+A secret is a long random string. Cross-platform:
 
 **Node (works everywhere Node is installed — the safe default here):**
 ```
@@ -199,17 +166,15 @@ node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
 
 **macOS / Linux:** `openssl rand -hex 32`
 
-Generate a *fresh* value per secret — never reuse one string for two variables,
-and never reuse the example's `change-me-...` placeholders.
+Never reuse the example's `change-me-...` placeholder.
 
 ## Common pitfalls
 
 - **Node too old** → `node:sqlite` import fails. Needs 22.5+.
-- **`SERVICE_TOKEN` ≠ `CRM_SERVICE_TOKEN`** → scraper gets 401, no leads land.
 - **Editing `.env.example` instead of `.env`** → app reads `.env`; the example is
   only a template. Never commit `.env`.
 - **Treating the `ExperimentalWarning` as a failure** → it's normal.
 - **Expecting Chat to work without an AI endpoint** → it's inert until Ollama (or
   a hosted endpoint) is reachable; the rest of the app is unaffected.
-- **Production boot refuses to start** → that's the fail-closed check; set real
-  `SESSION_SECRET` / `SETTINGS_KEY`, don't downgrade `NODE_ENV`.
+- **Saving AI/SMTP credentials fails in production** → that's the fail-closed
+  check; set a real `SETTINGS_KEY`, don't downgrade `NODE_ENV`.

.github/workflows/ci.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        pkg: [api, web, scraper]
+        pkg: [api, web]
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4

Dockerfile

@@ -1,6 +1,5 @@
 # syntax=docker/dockerfile:1
-# One image holds: the built internal web app, the API that serves it, and the
-# scraper. Build context is the crm/ directory.
+# One image holds the built internal web app and the API that serves it.
 
 # ---- Stage 1: build the internal web app (Vite) ----
 FROM node:24-alpine AS web
@@ -15,15 +14,12 @@ RUN npm run build
 FROM node:24-alpine AS runtime
 WORKDIR /app
 
-# Install deps (cached unless the lockfiles change).
+# Install deps (cached unless the lockfile changes).
 COPY api/package.json api/package-lock.json ./api/
 RUN cd api && npm ci
-COPY scraper/package.json scraper/package-lock.json ./scraper/
-RUN cd scraper && npm ci
 
 # App source.
 COPY api/ ./api/
-COPY scraper/ ./scraper/
 
 # Built web app, served by the API in production.
 COPY --from=web /web/dist ./web/dist
@@ -39,5 +35,4 @@ EXPOSE 8787
 # Liveness: the API exposes an unauthenticated /api/health.
 HEALTHCHECK --interval=30s --timeout=4s --start-period=20s --retries=3 \
   CMD node -e "fetch('http://127.0.0.1:'+(process.env.PORT||8787)+'/api/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"
-# Default command runs the API. The scraper service overrides this.
 CMD ["npm", "start"]

PROGRESS.md

@@ -4,6 +4,33 @@ A running log of what's landed, so picking the work back up is easy. Newest firs
 
 ## Latest
 
+- **Rewrite: reorganised, hardened, scraper removed, tailored to isarwebsites.**
+  **Structure:** the 1,500-line `api/src/index.ts` is now a slim composition root;
+  every endpoint moved into `api/src/routes/*` (one module per domain) with shared
+  `middleware.ts` (auth gates, proxy-aware client IP) and `helpers.ts` (one upload
+  validator instead of three copies, CSV/inline-file responses). **Security:**
+  sessions are now server-side DB rows (SHA-256 token hashes only) — logout,
+  password resets and user deletion revoke them for real, and `SESSION_SECRET` is
+  gone entirely; added `hono/csrf` origin checks on mutating requests,
+  `secure-headers` (+ CSP in prod), request-size limits (2 MB JSON / 12 MB uploads /
+  200 MB restore), login rate-limit keyed by real client IP (`TRUST_PROXY`), audited
+  `login.success`/`login.failed` with IP, and `/api/settings` is admin-only in both
+  directions. **Efficiency:** `GET /api/documents` no longer runs 2 queries per
+  document (bulk items + payments buckets), and document reads use an explicit
+  column list so signed-doc BLOBs never ride along in lists. **Removed:** the whole
+  `scraper/` workspace, `/api/scraper/*`, `scrape.ts`, the scraper settings/secret
+  columns (dropped by migration), the Scraper tab/view/types, `SERVICE_TOKEN`
+  (the xlsx CLI now writes via `insertLead` directly), the stale `mcp/` workspace
+  (targeted the removed `/api/v1`), and `docs/INTEGRATIONS.md` (documented removed
+  subsystems). **isarwebsites:** fresh installs seed an 11-item Leistungskatalog
+  (Website Starter/Business/Premium, Relaunch, Hosting & Pflege, SEO, Google
+  Business Profil, …) + default business name; copilot/analyst/outreach prompts
+  now sell websites for isarwebsites; UI rebranded (nav, login, title); README/
+  ROADMAP/DEPLOY/setup-skill rewritten. **125 API tests green**, api + web
+  typecheck clean, web builds; verified live in the browser (login, tabs, seeded
+  catalog under Einstellungen, draft invoice, dashboards) and via curl (401 gating,
+  form-CSRF 403, security headers, logout revocation).
+
 - **Streamline: cut Zeiterfassung + Customer-360, unify document saving.** Trimmed
   back toward a focused tool rather than a swiss-army knife. **Removed** the
   Zeiterfassung module (table, `timetracking.ts`, `/api/time*`, AI time tools,