Distributed Lease-Based Exactly-Once Execution Primitive

Go + Redis + Postgres. Guarantees an operation executes exactly once under concurrent retries, within a single Redis primary's serialized execution window.

Structure

.
├── cmd/server/main.go              # Smoke-test entry point
├── internal/
│   ├── lease/service.go            # Core API: CheckAndHold, Settle, Cancel, GetStatus
│   └── store/
│       ├── redis.go                # Lua script loading + Redis calls
│       └── postgres.go             # Durable settled outcomes
├── scripts/
│   ├── check_and_hold.lua          # Atomic hold acquisition
│   ├── cancel.lua                  # Holder-safe hold release
│   └── mark_settled.lua            # Cache update after Postgres write
├── migrations/
│   └── 001_create_settled_outcomes.sql
├── test/
│   └── integration_test.go         # The one invariant test
└── docker-compose.yml              # Real Redis + Postgres

Run the integration test

docker compose up -d
# Wait ~5 seconds for healthchecks to pass, then:
go test -v -race -count=1 ./test/

Design decisions — where to find the reasoning

Decision	Where it's explained
Lua vs MULTI/EXEC	scripts/check_and_hold.lua header
Postgres-before-Redis write order	internal/store/postgres.go WriteSettledOutcome() comment
Why TTL and not a heartbeat	internal/lease/service.go Config struct comments
Cancel safety (holder ID check)	scripts/cancel.lua header
EVALSHA + NOSCRIPT recovery	internal/store/redis.go evalSHA()
ON CONFLICT DO NOTHING in Postgres	internal/store/postgres.go WriteSettledOutcome()
GetStatus read-repair path	internal/lease/service.go GetStatus()

What the integration test checks

TestExactlyOnceUnderConcurrentRetries: 50 goroutines race for the same key.

• Exactly 1 acquires the hold (Redis Lua atomicity)
• Exactly 1 calls Settle (application-level exactly-once)
• Exactly 1 Postgres row exists (durability + ON CONFLICT protection)
• GetStatus returns StatusSettled with valid result (read path)

Edges of the "exactly once" claim

Say these unprompted in interviews:

1. Failover window: async Redis replication → holds written just before primary death may not exist on the promoted replica. Fix: WAIT or synchronous replication. Otherwise: document as "exactly once except during the failover window."

2. TTL miscalibration: if DefaultHoldTTL < p99_operation_latency, the lease expires mid-execution and a second caller can acquire. Calibrate from real measurements.

3. Redis Cluster cross-slot: keys for the same logical resource must hash to the same slot. Fix: hash tags — {resource_id}:hold and {resource_id}:settled share {resource_id} as the hash tag.