[Proposal] High-Scale Workspace Isolation & Multi-Tenancy Architecture (v1.4.10)

[djrfree]

Executive Summary:

I have implemented a comprehensive workspace isolation layer on top of v1.4.10 (build 0272) to enable true multi-tenant operations. This architecture moves away from a single global state to a managed, per-request RAG instance model. It has been stress-tested using PostgreSQL (pgvector + Apache AGE) in a high-scale environment requiring a large number of independent workspaces.

Core Architectural Changes:

RAGInstanceManager:

A new management layer using asyncio double-check locking to ensure thread-safe, concurrent initialization of workspace-specific instances.

Intelligent Caching:

Implemented LRU (Least Recently Used) eviction and TTL-based cleanup to manage memory and database connection pools across "unlimited" potential workspaces.

Dynamic Routing:

Workspace selection is now malleable per-request via the LIGHTRAG-WORKSPACE HTTP header or request body parameters, overriding the server's default .env configuration.

Full-Stack Support:

Web UI: Added a persistent workspace selector in the settings panel (0272) that injects headers into all axios and fetch calls.
Integrations: Extended the Open WebUI tool with workspace valve configurations.

Critical Fixes Included:

Resolved PostgreSQL serialization issues where bytes/memoryview objects failed during JSON-compatible string conversion.
Corrected the paginated documents endpoint to use workspace-aware parameters and proper Pydantic field typing.
Fixed the pipeline status endpoint to retrieve context-specific RAG instances.

Discussion Goal:

I believe this architecture provides a production-grade foundation for LightRAG multi-tenancy. I would like to discuss:

• Is there interest in adopting this RAGInstanceManager pattern into the official API?
• Should the PostgreSQL serialization and Pydantic fixes be submitted as a standalone "pre-requisite" PR?
• I am happy to share the code or prepare a PR based on the community's feedback.

[boomerman]

I have this requirement. I am also developing a multitenant multiuser application.

[djrfree]

It appears that the development team is actively working on the upgrade to version 2, so I am reluctant to submit a PR right now. If version 2 doesn't contain something like this, I am prepared to rework the code for the new version.