A production-grade, drop-in etcd replacement backed by Google Cloud Spanner.
Implements the complete etcd v3 KV/Watch/Lease/Auth gRPC API. Deploy spanner-etcd in place of etcd and gain unlimited horizontal scale, native multi-region replication, and 99.999% availability — with zero etcd cluster management overhead.
# Before
--etcd-servers=https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379
# After
--etcd-servers=http://spanner-etcd:2379
| etcd | spanner-etcd | |
|---|---|---|
| Storage | Raft log on local disk | Google Cloud Spanner |
| Horizontal scale | Limited (3–5 members) | Unlimited stateless replicas |
| Cross-region | Manual federation | Native (Spanner multi-region) |
| Durability | Single-region by default | 99.999% SLA |
| Operations | Cluster management, backups | Fully managed by Google |
| Write throughput (×32) | baseline | 15× faster (PENDING_COMMIT_TIMESTAMP) |
| Watch latency | ~1ms (local) | ~30ms (Change Streams) |
Standard etcd is a single-region, single-cluster system. Under sustained load it becomes a write bottleneck — a global counter serializes every mutation. The GKE team replaced etcd with a Spanner-backed system to scale Kubernetes clusters to 65,000+ nodes.
spanner-etcd is an open implementation of the same idea:
- PENDING_COMMIT_TIMESTAMP as revision — no shared counter, no write lock, each transaction is fully independent
- Spanner Change Streams for Watch — push-based delivery at ~10–50ms latency
- Atomic Txn — compare+ops in a single Spanner ReadWriteTransaction
- Stateless replicas — all state in Spanner, scale out by adding pods
# Start Spanner emulator
docker run -d -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator
# Create instance and database
curl -s -X POST http://localhost:9020/v1/projects/my-project/instances \
-d '{"instanceId":"my-instance","instance":{"config":"emulator-config","displayName":"dev","nodeCount":1}}'
curl -s -X POST http://localhost:9020/v1/projects/my-project/instances/my-instance/databases \
-d '{"createStatement":"CREATE DATABASE `my-db`"}'
# Run spanner-etcd
SPANNER_EMULATOR_HOST=localhost:9010 spanner-etcd \
--spanner-database=projects/my-project/instances/my-instance/databases/my-db
# Test
etcdctl put /hello world && etcdctl get /hello| Doc | Contents |
|---|---|
| Architecture | Design decisions, Spanner schema, etcd API coverage, known limitations |
| Deployment | GKE/Helm, kubeadm, TLS, migration, monitoring |
| Configuration | All flags, Workload Identity Federation, Helm values |
| Performance | Benchmarks, production validation, soak test results |
| Development | Build, test, CI, Makefile targets |
| Component | Status |
|---|---|
| etcd v3 KV/Watch/Lease/Auth API | ✅ Complete |
| Atomic Txn (single Spanner RWT) | ✅ Complete |
| Spanner Change Streams | ✅ Production Spanner |
| Background auto-compaction | ✅ Complete |
| Prometheus metrics + GKE PodMonitoring | ✅ Complete |
| Helm chart (WIF, PDB, HPA) | ✅ Complete |
| 78 integration tests | ✅ Passing |
| Kubernetes v1.33.12 (kubeadm) | ✅ Validated |
| 24h soak test (Kubernetes v1.33) | ✅ Complete |
| TLS / mTLS | ✅ Validated |
| Multi-replica HA | ✅ Validated |
Apache 2.0