feat: DB inbound path (Toolbox config generator), MCP server fixes, v… by gipsbrian · Pull Request #2 · iblai/ontology

gipsbrian · 2026-07-03T18:04:04Z

Checklist

Tests were added/updated according to the feature/bugfix/change made
Version was rolled according to semver requirements
API endpoints openapi schema was updated if applicable

Changes

Makes the on-prem knowledge layer actually connect to real backends and cleans up the connector story so clients can point it at their own Canvas, PostgreSQL, and MySQL. Adds a DSL to Google MCP Toolbox config generator, fixes the custom MCP servers, ships generic backend-neutral database connectors, and separates deployment-specific wiring from the committed repo.

Branch: setup/bringup-debug -> main. Version stays 0.2.0 (unreleased).

Why

The database inbound path was scaffolding: the shipped docker-compose.yml never actually started the Toolbox (bad command), the repo's tools.yaml DSL is not the format the real Toolbox consumes, the gateway proxied to a Toolbox endpoint that is disabled in current versions, and all four custom MCP servers used an invalid mcp.server.Server API. This branch makes the whole path work end-to-end and generic.

What's included

Inbound: database path

DSL to Toolbox config generator (backend/mcp_server/toolbox_config.py): translates the authoring DSL (kind: source|tool|toolset, type:) into the Toolbox-native sources/tools/toolsets maps at config/generated/toolbox.yaml. Maps type->kind, renames Oracle database->serviceName, preserves ${VAR} tokens (secrets never written to disk), and excludes ${...} raw-injection tools as gateway-native.
New CLI ontology mcp build; ontology deploy up regenerates it automatically.
Gateway proxy now calls the Toolbox /mcp JSON-RPC endpoint (the legacy /api/tool/<name> REST path is disabled in Toolbox >= 1.5).
Generic, backend-neutral connectors (connect to any local or remote DB by host + read-only credentials):
- client-postgres (CLIENT_POSTGRES_*) with list-postgres-tables / describe-postgres-table.
- client-mysql (CLIENT_MYSQL_*) with list-mysql-tables / describe-mysql-table (MySQL ? placeholders + information_schema).

Inbound: custom API MCP servers

Migrated canvas, slate, navigate, ldap from the invalid Server/@server.tool() usage to FastMCP.
Canvas: resolves users by Canvas user id, SIS id, or sis_login_id: (was SIS-only); get_student_submissions uses the real per-course submissions endpoint and calls raise_for_status().

Config and deployment

docker-compose.yml: Toolbox loads the generated config via --config; fixed the duplicated toolbox entrypoint arg. Deployment-specific external source networks were removed from the committed compose (a fresh clone now parses without pre-existing networks).
Source-network wiring moved to a git-ignored docker-compose.override.yml, with a committed docker-compose.override.example.yml showing Pattern A (local source container network, connect by hostname) and Pattern B (egress network for a remote source, connect by host/IP).
Higher-ed PeopleSoft sample split into optional config/tools.higher-ed.example.yaml + config/roles.higher-ed.example.yaml, so the default stack starts clean (the Toolbox eagerly connects to every source at startup).
Per-service .env.*.example templates; .gitignore updated so templates are tracked while real .env*, the override, and .python-version stay ignored.

Docs and tests

New docs/read-only-db-user.md (read-only Postgres + MySQL user provisioning); updated docs/deployment.md and docs/components/01-mcp-inbound.md; new CHANGELOG.md.
Generator unit tests; updated repo-config/identity tests for the new baseline. Full suite: 169 passing.

Validation

Exercised end-to-end against live databases (not just unit tests):

The real MCP Toolbox, loaded from the generated config, ran parameterized tools over /mcp against a live PostgreSQL and a live MySQL and returned real rows.
Read-only DB roles verified: read succeeds, all writes denied.
Canvas tools driven against a live Canvas instance.
Base docker-compose.yml parses standalone (portable); base + override merges correctly.

Notes and follow-ups (not in this PR)

config/initializer.py (used by ontology config init) still scaffolds the full higher-ed sample.
There is no default admin dashboard; the admin-dashboard service referenced in compose has no build context yet.
Full-stack deploy up (all services together) has not been run end-to-end.

Security

No source writes: read-only everywhere; generated config keeps ${VAR} tokens so credentials are never written to disk.
Real secrets and deployment-specific overrides are git-ignored; only .example templates are committed.

…0.2.0 Bring up the ontology stack end-to-end and make the database inbound path actually functional against the real Google MCP Toolbox. - Add DSL -> Toolbox-native config generator (backend/mcp_server/toolbox_config.py) + `ontology mcp build`; `ontology deploy up` regenerates it automatically. Maps type->kind, oracle database->serviceName, preserves ${VAR} (secrets stay off disk), excludes ${...} raw-injection tools as gateway-native. - Fix gateway _proxy_to_toolbox to use the Toolbox /mcp JSON-RPC endpoint (legacy /api/tool/<name> is disabled in Toolbox >=1.5). - Migrate custom MCP servers (canvas/slate/navigate/ldap) to FastMCP; Canvas resolves by Canvas id / SIS id / sis_login_id and uses the real per-course submissions endpoint. - Add generic env-driven client-postgres source + parameterized list-client-tables / describe-client-table tools + client-db-tools toolset. - Split the higher-ed sample into config/tools.higher-ed.example.yaml and config/roles.higher-ed.example.yaml so the default stack starts clean (the Toolbox eagerly connects to every source at startup). - docker-compose: load generated config via --config, fix duplicated toolbox arg. - Add docs/read-only-db-user.md, per-service .env.*.example templates, CHANGELOG. - Tests for the generator; update repo-config/identity tests for the new baseline. - Bump version to 0.2.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add MySQL support and make the DB connectors client-neutral so clients don't read them as backend-specific. - Add generic env-driven `client-mysql` source (CLIENT_MYSQL_*) + mysql-sql tools list-mysql-tables / describe-mysql-table (? placeholders, information_schema) + client-mysql-tools toolset. No generator change needed — MySQL is a first-class Toolbox source type. - Rename the Postgres connector for symmetry/neutrality: CLIENT_DB_* -> CLIENT_POSTGRES_*, list-client-tables -> list-postgres-tables, describe-client-table -> describe-postgres-table, client-db-tools -> client-postgres-tools. - Scrub deployment-specific names from tracked files (edx/openedx/ibl_dm_db/ dlmanager/ibl_local/ibleducation) -> generic placeholders; both connectors connect to any local or remote Postgres/MySQL by host + credentials. - Move source-network wiring out of the committed docker-compose.yml into a git-ignored override (add docker-compose.override.example.yml with a local and a remote pattern) so a fresh clone parses without pre-existing networks. - Docs: generic MySQL read-only user section + source-networking guidance. No version bump (branch not yet merged to main). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Fixes raised in the setup/bringup-debug self-review before merge. - M1: extract a shared Toolbox /mcp client (backend/mcp_server/toolbox_client.py) and route BOTH the gateway (handlers._proxy_to_toolbox) and the sync engine (sync.engine.pull) through it. The sync engine was still calling the legacy /api/tool/<name> endpoint (disabled in Toolbox >=1.5) with the old response shape; now both callers use /mcp JSON-RPC and can't drift. - M2: Canvas get_student_submissions no longer aborts all courses when one course returns 403/concluded — per-course failures are skipped. - M3: Canvas user lookup no longer treats a bare numeric value as SIS-vs-Canvas ambiguously; a bare value is a Canvas user id and SIS ids must be passed as sis_user_id:<id> (they are often numeric). Docstrings/tool descriptions updated. - L1: `ontology deploy up` reports a clean error instead of a traceback when toolbox-config generation fails. - L2: removed the redundant local json import in the gateway proxy. - L3: MySQL list/describe tools alias columns so keys are lowercase (table_name/column_name), matching the Postgres tools. - L4: repo-config guard test now also requires client-mysql-tools. - L5: Canvas list endpoints follow Link rel=next pagination (no silent truncation). - Tests: new toolbox_client tests; updated sync pull test for the /mcp path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gipsbrian · 2026-07-03T19:02:26Z

Testing

Automated (`pytest`)

173 passed on Python 3.11.10. New/updated coverage in this branch:

tests/test_toolbox_config.py â�� DSL to Toolbox config generator (type->kind, Oracle database->serviceName, ${...} native-tool exclusion, toolset pruning).
tests/test_toolbox_client.py â�� shared /mcp client (JSON + SSE parsing, JSON-RPC error, transport error).
tests/test_sync.py â�� sync pull() now asserts the /mcp JSON-RPC path and row unwrapping.
tests/test_repo_config.py, tests/test_toolbox_compat.py, tests/test_identity.py, tests/test_service_add_and_middleware.py â�� updated for the trimmed baseline config and merged higher-ed sample roles.

Live end-to-end

All three connectors were driven against real backends after the final changes:

Component	Exercised	Result
Custom MCP servers (canvas/slate/navigate/ldap)	Import under `FastMCP`	Pass
Canvas	`get_student_courses` / `get_student_submissions` against a live Canvas instance (admin token)	Pass â�� returns real enrollment; per-course failures skipped; no crash
PostgreSQL (`client-postgres`)	`list-postgres-tables` / `describe-postgres-table` via the real Toolbox over `/mcp` against a live DB (~647 tables)	Pass â�� real rows returned
MySQL (`client-mysql`)	`list-mysql-tables` / `describe-mysql-table` via the real Toolbox over `/mcp` against a live DB (~540 tables)	Pass â�� real rows, keys lowercased to match Postgres
Read-only DB roles	`SELECT` succeeds; `CREATE TABLE` attempt	Pass â�� writes denied on both Postgres and MySQL
Generated Toolbox config	Loaded by Google MCP Toolbox `1.5.0` and tools invoked over `/mcp`	Pass
Compose portability	Base `docker-compose.yml` parses standalone; `+ override` merges	Pass

Notes on the /mcp migration: both callers (the gateway handlers._proxy_to_toolbox and the sync engine pull()) now go through the shared toolbox_client, so the legacy /api/tool/<name> endpoint (disabled in Toolbox >= 1.5) is no longer used anywhere.

Not covered (tracked follow-ups)

Full-stack ontology deploy up (all services together) has not been run end-to-end.
The Canvas MCP server is validated live rather than via pytest (it runs as a standalone container).
config/initializer.py still scaffolds the higher-ed sample for ontology config init.

One-time normalization so the ruff CI (added next) passes repo-wide: - ruff format (0.14.14) applied to all Python files. - Import sorting (ruff check --select I) applied. - Removed a few unused imports / one unused local flagged by ruff check. No behavior changes; 173 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Mirrors the data-manager workflow: pins ruff 0.14.14, checks `format --diff` and `check --select I` repo-wide, and lints changed files on every PR to main. Adds ruff.toml (target py311; excludes migrations + config/generated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds a Tests workflow: sets up Python 3.11, installs the package with the dev + django extras (what the suite needs), and runs `pytest` on PRs to main so the 173-test suite gates merges. Verified via a clean-venv install + run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`./dev.sh check` runs the same checks as CI (ruff format/import-sort/lint via the pinned 0.14.14, then pytest) so failures are caught before committing. Also `setup`, `test [args]`, `fmt`, `lint`. Documented under README "Local checks". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gipsbrian requested a review from mamigot July 3, 2026 18:08

gipsbrian and others added 2 commits July 3, 2026 18:45

gipsbrian and others added 4 commits July 3, 2026 19:10

gipsbrian merged commit 7cb693e into main Jul 3, 2026
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: DB inbound path (Toolbox config generator), MCP server fixes, v…#2

feat: DB inbound path (Toolbox config generator), MCP server fixes, v…#2
gipsbrian merged 7 commits into
mainfrom
setup/bringup-debug

gipsbrian commented Jul 3, 2026 •

edited

Loading

Uh oh!

gipsbrian commented Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

gipsbrian commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Changes

Why

What's included

Inbound: database path

Inbound: custom API MCP servers

Config and deployment

Docs and tests

Validation

Notes and follow-ups (not in this PR)

Security

Uh oh!

gipsbrian commented Jul 3, 2026

Testing

Automated (pytest)

Live end-to-end

Not covered (tracked follow-ups)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gipsbrian commented Jul 3, 2026 •

edited

Loading

Automated (`pytest`)