Skip to content

refactor: add support for postgres store(s) #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
frrist wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

refactor: add support for postgres store(s) #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,11 @@ Sprue is the upload coordination service for Storacha local development. It rout

**Stores (pkg/store/)**
- Each domain has its own store interface in `pkg/store/<domain>/`
- Each store has two implementations: AWS (DynamoDB/S3) in `<domain>/aws/` and in-memory in `<domain>/memory/`
- Each store has three implementations: AWS (DynamoDB/S3) in `<domain>/aws/`, PostgreSQL (+ S3 for blob payloads) in `<domain>/postgres/`, and in-memory in `<domain>/memory/`
- Store interfaces: `agent.Store`, `blob_registry.Store`, `consumer.Store`, `customer.Store`, `delegation.Store`, `metrics.Store`, `replica.Store`, `revocation.Store`, `space_diff.Store`, `storage_provider.Store`, `subscription.Store`, `upload.Store`
- AWS stores are wired in `internal/fx/store/aws/provider.go`, memory stores in `internal/fx/store/memory/provider.go`
- Backends are wired in `internal/fx/store/<backend>/provider.go` (aws, postgres, memory)
- Backend selection is driven by `storage.type` in config (`memory` | `postgres` | `aws`; default `postgres`). Per-backend settings live under `storage.postgres`, `storage.dynamodb`, and `storage.s3`.
- Postgres schema is managed by goose migrations in `internal/migrations/sql/`, embedded and applied on startup. Set `storage.postgres.skip_migrations: true` to disable.

**Services (pkg/)**
- `provisioning`: Manages space provisioning (consumers + subscriptions)
Expand All @@ -66,18 +68,21 @@ Sprue is the upload coordination service for Storacha local development. It rout

### Configuration

Configuration via YAML file or environment variables with `UPLOAD_` prefix:
- `UPLOAD_SERVER_HOST`, `UPLOAD_SERVER_PORT`
- `UPLOAD_IDENTITY_KEY_FILE`, `UPLOAD_IDENTITY_PRIVATE_KEY`, `UPLOAD_IDENTITY_SERVICE_DID`
- `UPLOAD_PIRI_ENDPOINT`, `UPLOAD_INDEXER_ENDPOINT`
- `UPLOAD_DYNAMODB_*` for DynamoDB settings

Legacy env vars without prefix (e.g., `HOST`, `PORT`, `KEY_FILE`) also supported.
Configuration via YAML file or environment variables with `SPRUE_` prefix:
- `SPRUE_STORAGE_TYPE` — selects the store backend (`memory`, `postgres`, `aws`; default `postgres`)
- `SPRUE_SERVER_HOST`, `SPRUE_SERVER_PORT`
- `SPRUE_IDENTITY_KEY_FILE`, `SPRUE_IDENTITY_PRIVATE_KEY`, `SPRUE_IDENTITY_SERVICE_DID`
- `SPRUE_INDEXER_ENDPOINT`
- `SPRUE_STORAGE_POSTGRES_DSN`, `SPRUE_STORAGE_POSTGRES_MAX_CONNS`, `SPRUE_STORAGE_POSTGRES_SKIP_MIGRATIONS`
- `SPRUE_STORAGE_DYNAMODB_*` for DynamoDB settings (AWS backend)
- `SPRUE_STORAGE_S3_*` for S3/MinIO settings

### Key Dependencies

- **go-ucanto**: UCAN RPC framework for capability-based authorization
- **go-libstoracha**: Storacha capability definitions (blob, space, upload, etc.)
- **echo/v4**: HTTP server framework
- **aws-sdk-go-v2**: DynamoDB client
- **aws-sdk-go-v2**: DynamoDB + S3 client
- **jackc/pgx/v5**: PostgreSQL driver
- **pressly/goose/v3**: SQL schema migrations
- **viper/cobra**: Configuration and CLI
26 changes: 25 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,30 @@
# Sprue

The Storacha upload service in Go.
The Forge upload service in Go (formerly the Storacha upload service).

## Running locally

The repo ships a `docker-compose.yaml` that brings up sprue alongside
PostgreSQL and MinIO for self-hosted development:

```bash
docker compose up -d postgres minio
SPRUE_STORAGE_POSTGRES_DSN="postgres://sprue:sprue@localhost:5432/sprue?sslmode=disable" \
./sprue serve
```

Postgres is the default store backend, so no extra flag is required.

## Store backends

Sprue supports three store backends, selected by
`storage.type` (or `SPRUE_STORAGE_TYPE`; defaults to `postgres`):

- `memory` — in-process only; all data is lost on restart. Dev/test only.
- `postgres` — PostgreSQL for metadata + S3-compatible storage (MinIO, Ceph, AWS S3)
for blob payloads. Schema is managed by goose migrations embedded in
`internal/migrations/sql/` and applied on startup.
- `aws` — DynamoDB for metadata + S3 for blob payloads.

## Notes

Expand Down
82 changes: 48 additions & 34 deletions config.example.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# Sprue Configuration
#
# This file documents all configuration options. Copy to config.yaml and customize.
# All values can also be set via environment variables with UPLOAD_ prefix.
# All values can also be set via environment variables with SPRUE_ prefix.
# Environment variables take precedence over config file values.
#
# Examples:
# server.host -> UPLOAD_SERVER_HOST
# dynamodb.endpoint -> UPLOAD_DYNAMODB_ENDPOINT
# server.host -> SPRUE_SERVER_HOST
# storage.type -> SPRUE_STORAGE_TYPE
# storage.postgres.dsn -> SPRUE_STORAGE_POSTGRES_DSN
#

# Deployment configuration
Expand All @@ -20,11 +21,6 @@ deployment:
# includes the original blob that was uploaded, so only values above 1 will
# allow users to have multiple copies of their data.
max_replicas: 3
# Indicates whether to use in-memory stores instead of DynamoDB/S3. All data
# will be lost on service restart when this is true, so it should only be used
# for development or testing. It overrides all other store-related config when
# true.
in_memory_stores: false

server:
# Host address to bind the HTTP server to
Expand Down Expand Up @@ -63,33 +59,51 @@ mailer:
# Secret for CRAMMD5 SMTP authentication
smtp_auth_secret: ""

dynamodb:
# DynamoDB endpoint (for local development with DynamoDB Local)
endpoint: "http://dynamodb-local:8000"
# AWS region for DynamoDB
region: "us-west-1"
agent_index_table: "agent-index"
blob_registry_table: "blob-registry"
consumer_table: "consumer"
customer_table: "customer"
delegation_table: "delegation"
space_metrics_table: "space-metrics"
admin_metrics_table: "admin-metrics"
replica_table: "replica"
revocation_table: "revocation"
storage_provider_table: "storage-provider"
subscription_table: "subscription"
space_diff_table: "space-diff"
upload_table: "upload"
# Storage backend selection and per-backend configuration.
storage:
# Selects the backend. Valid values: "memory", "postgres", "aws". Default "postgres".
# - memory: in-process stores — all data lost on restart; dev/test only.
# - postgres: PostgreSQL for metadata + S3-compatible storage for blob payloads.
# - aws: DynamoDB for metadata + S3 for blob payloads.
type: "postgres"

s3:
# S3 endpoint (for local development with MinIO)
endpoint: "http://minio:9000"
# S3 region
region: "us-west-1"
agent_message_bucket: "agent-message"
delegation_bucket: "delegation"
upload_shards_bucket: "upload-shards"
postgres:
# libpq-style connection string (used when storage.type is "postgres").
dsn: "postgres://sprue:sprue@postgres:5432/sprue?sslmode=disable"
# Maximum number of pool connections. 0 uses the pgx default.
max_conns: 10
# Minimum idle connections to keep warm.
min_conns: 0
# Set to true to skip goose migrations at startup (they run by default).
skip_migrations: false

dynamodb:
# DynamoDB endpoint (for local development with DynamoDB Local).
endpoint: "http://dynamodb-local:8000"
# AWS region for DynamoDB.
region: "us-west-1"
agent_index_table: "agent-index"
blob_registry_table: "blob-registry"
consumer_table: "consumer"
customer_table: "customer"
delegation_table: "delegation"
space_metrics_table: "space-metrics"
admin_metrics_table: "admin-metrics"
replica_table: "replica"
revocation_table: "revocation"
storage_provider_table: "storage-provider"
subscription_table: "subscription"
space_diff_table: "space-diff"
upload_table: "upload"

s3:
# S3 endpoint (for local development with MinIO).
endpoint: "http://minio:9000"
# S3 region.
region: "us-west-1"
agent_message_bucket: "agent-message"
delegation_bucket: "delegation"
upload_shards_bucket: "upload-shards"

log:
# Log level: debug, info, warn, error
Expand Down
53 changes: 33 additions & 20 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,41 @@ require (
github.com/ipfs/go-cid v0.6.0
github.com/ipfs/go-log/v2 v2.9.0
github.com/ipld/go-ipld-prime v0.21.1-0.20240917223228-6148356a4c2e
github.com/jackc/pgx/v5 v5.9.1
github.com/labstack/echo/v4 v4.14.0
github.com/multiformats/go-multiaddr v0.16.0
github.com/multiformats/go-multibase v0.2.0
github.com/multiformats/go-multihash v0.2.3
github.com/olekukonko/tablewriter v0.0.5
github.com/pressly/goose/v3 v3.27.0
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/storacha/go-libstoracha v0.7.5
github.com/storacha/go-ucanto v0.7.2
github.com/stretchr/testify v1.11.1
github.com/testcontainers/testcontainers-go v0.41.0
github.com/testcontainers/testcontainers-go v0.42.0
github.com/testcontainers/testcontainers-go/modules/dynamodb v0.41.0
github.com/testcontainers/testcontainers-go/modules/minio v0.40.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
go.uber.org/fx v1.24.0
go.uber.org/zap v1.27.0
)

require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/mfridman/interpolate v0.0.2 // indirect
github.com/moby/moby/api v1.54.1 // indirect
github.com/moby/moby/client v0.4.0 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/sethvargo/go-retry v0.3.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect
go.opentelemetry.io/otel/sdk v1.43.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.43.0 // indirect
golang.org/x/sync v0.20.0 // indirect
)

require (
dario.cat/mergo v1.0.2 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
Expand Down Expand Up @@ -59,7 +77,6 @@ require (
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/creack/pty v1.1.24 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
Expand All @@ -78,7 +95,6 @@ require (
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
Expand All @@ -99,7 +115,7 @@ require (
github.com/ipld/go-car v0.6.2 // indirect
github.com/ipld/go-codec-dagpb v1.6.0 // indirect
github.com/ipni/go-libipni v0.6.18 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
github.com/labstack/gommon v0.4.2 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
Expand All @@ -112,12 +128,11 @@ require (
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.2.0 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/patternmatcher v0.6.1 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-base32 v0.1.0 // indirect
github.com/multiformats/go-base36 v0.2.0 // indirect
Expand All @@ -136,8 +151,8 @@ require (
github.com/polydawn/refmt v0.89.1-0.20231129105047-37766d95467a // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/sagikazarmark/locafero v0.11.0 // indirect
github.com/shirou/gopsutil/v4 v4.26.2 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/shirou/gopsutil/v4 v4.26.3 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/afero v1.15.0 // indirect
Expand All @@ -153,24 +168,22 @@ require (
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
go.opentelemetry.io/otel v1.42.0 // indirect
go.opentelemetry.io/otel/metric v1.42.0 // indirect
go.opentelemetry.io/otel/trace v1.42.0 // indirect
go.opentelemetry.io/otel v1.43.0 // indirect
go.opentelemetry.io/otel/metric v1.43.0 // indirect
go.opentelemetry.io/otel/trace v1.43.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/dig v1.19.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa // indirect
golang.org/x/net v0.51.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/crypto v0.49.0 // indirect
golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa // indirect
golang.org/x/net v0.52.0 // indirect
golang.org/x/sys v0.42.0 // indirect
golang.org/x/text v0.35.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260316172706-e463d84ca32d // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260311181403-84a4fc48630c // indirect
google.golang.org/grpc v1.79.2 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.4.1 // indirect
Expand Down
Loading
Loading