Status: Draft
Authors:
This RFC proposes a catalog system for OpenData that serves as a central management plane for OpenData storage systems. The catalog provides a single point to manage metadata about the systems ("slates") a user has installed, including their names, types, and object storage configuration. The catalog itself is implemented as a slate backed by SlateDB, following the same patterns as other OpenData subsystems.
OpenData comprises multiple storage subsystems (log, timeseries, vector, etc.) that share a common foundation on SlateDB. As users deploy multiple instances of these systems, several cross-cutting operational concerns emerge:
-
Discovery — There is no unified way to enumerate which slates exist within an environment. Operators must track this information externally or inspect object storage directly.
-
Configuration management — Each slate requires object storage configuration (bucket, path prefix, credentials). Without a central registry, this configuration is scattered and difficult to audit.
-
Lifecycle management — Creating and deleting slates involves coordinating metadata across multiple locations. A catalog provides a single point of control for these operations.
-
Operational tooling — Cross-cutting tools (backup, monitoring, migration) need a way to discover and enumerate slates without subsystem-specific knowledge.
A catalog addresses these concerns by providing:
- A single source of truth for slate metadata
- Consistent registration and deletion workflows
- A foundation for future operational tooling that works across subsystem types
The catalog is itself a slate backed by SlateDB, ensuring it benefits from the same durability and operational characteristics as the systems it manages.
$ opendata slate list
NAME TYPE BUCKET
events log s3://acme-data/events
metrics timeseries s3://acme-data/metrics
$ opendata slate register --name orders --type log --bucket s3://acme-data/orders
Registered slate 'orders'
$ opendata slate list
NAME TYPE BUCKET
events log s3://acme-data/events
metrics timeseries s3://acme-data/metrics
orders log s3://acme-data/orders
$ opendata slate describe orders
Name: orders
Type: log
Bucket: s3://acme-data/orders
$ opendata slate delete orders
Deleted slate 'orders'
$ opendata slate config events
storage:
type: SlateDb
path: events
object_store:
type: Aws
region: us-west-2
bucket: acme-data
- Define "slate" as the fundamental unit of an OpenData system
- Specify the metadata required to describe a slate (name, type, object storage location)
- Establish the catalog as a SlateDB-backed registry of slate metadata
- Define the registration process for adding new slates to the catalog
- Define the deletion process for removing slates from the catalog
- Ensure the catalog can manage slates of all types (log, timeseries, vector)
To be completed in a future revision.
The catalog does not operate as a persistent service with a communication endpoint. Instead, all interaction with the catalog occurs through its SlateDB-backed storage. This follows directly from SlateDB's Reader/Writer model, where components temporarily assume one of two roles:
- Writer — A component that opens the catalog with write access to mutate state (register slates, update metadata, record status changes).
- Reader — A component that opens a read-only view of the catalog to observe state changes.
This model enables coordination between loosely-coupled components without requiring a running service or explicit RPC. Components communicate implicitly by writing state that other components read.
Consider a flow where a user registers a new slate and a provisioning system responds:
- A CLI tool temporarily assumes the Writer role to register a new slate in the catalog.
- A provisioning system (e.g., a Kubernetes operator) running as a Reader observes the new registration.
- The provisioning system creates the corresponding infrastructure (pods, services, etc.).
- The provisioning system temporarily assumes the Writer role to update the slate's status, indicating provisioning is complete.
- The CLI (or other tooling) as a Reader can observe the updated status.
┌─────────┐ ┌─────────────────┐ ┌─────────────────┐
│ CLI │ │ Catalog │ │ Provisioner │
│ │ │ (SlateDB) │ │ (K8s, etc.) │
└────┬────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
│ [Writer] register │ │
│ slate "orders" │ │
│──────────────────────►│ │
│ │ │
│ │ [Reader] observe new │
│ │ registration │
│ │◄──────────────────────────│
│ │ │
│ │ create infra
│ │ │
│ │ [Writer] update status │
│ │ "provisioned" │
│ │◄──────────────────────────│
│ │ │
│ [Reader] observe │ │
│ status change │ │
│◄──────────────────────│ │
│ │ │
This is not a formal provisioning protocol—specific workflows are left for future work. The key insight is that the catalog's storage layer is the communication channel. There is no separate messaging system or service API.
- No always-on service — The catalog does not require a continuously running process. Components open Reader or Writer handles as needed.
- Distributed coordination via storage — Object storage (S3, GCS, etc.) serves as the durable communication medium.
- Consistency from SlateDB — SlateDB's single-writer guarantee ensures catalog mutations are serialized. Readers see a consistent snapshot.
- Polling for changes — Readers must poll to observe updates. Future work may explore change notification mechanisms built on SlateDB.
Additional design sections to be completed in a future revision.
To be completed in a future revision.
-
Catalog bootstrap — How is the catalog itself discovered? If the catalog is a slate, where is its object storage configuration stored?
-
Bucket metadata — What specific fields are required for object storage configuration? (bucket name, region, path prefix, credentials reference?)
-
Catalog location — Should there be one catalog per bucket, per region, or per "environment"? What is the deployment topology?
| Date | Description |
|---|---|
| 2026-01-21 | Initial draft |