|
| 1 | +--- |
| 2 | +id: cassandra-single-writer-design |
| 3 | +title: Cassandra single-writer design |
| 4 | +sidebar_label: Cassandra single-writer design |
| 5 | +--- |
| 6 | + |
| 7 | +Design notes for the compare-and-set snapshot mode of `kafka-flow-persistence-cassandra` |
| 8 | +(`CassandraSnapshots.withSchema(compareAndSet = true)`) — the mechanism and the reasoning around it. What |
| 9 | +ships is small: a single conditional predicate on each `persist`, with `delete` left unchanged. The |
| 10 | +substance here is why that suffices, and — separately — the delete fence that was designed, TLA+-modelled, |
| 11 | +and deferred (the *Conditional deletes* appendix, none of which ships). The Kafka backend solves the same |
| 12 | +problem differently — see [Kafka single-writer design](kafka-single-writer-design.md). |
| 13 | + |
| 14 | +## Problem |
| 15 | + |
| 16 | +[kafka-flow#732](https://github.com/evolution-gaming/kafka-flow/issues/732): during a rebalance a |
| 17 | +previous owner that has not yet observed the revocation keeps flushing snapshots alongside the new |
| 18 | +owner, and the last-write-wins snapshots table lets a stale write overwrite a newer one — the next |
| 19 | +recovery then loads stale state and skips the events in between. The |
| 20 | +[Kafka design doc](kafka-single-writer-design.md) covers the failure in full. |
| 21 | + |
| 22 | +## Mechanism: compare-and-set |
| 23 | + |
| 24 | +Cassandra has no transaction to bind the input-offset commit to, the way the Kafka backend does — but |
| 25 | +it does offer a conditional write (a Paxos lightweight transaction), so the fence is **per write**. The |
| 26 | +stored offset is the per-key [fencing token](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html): |
| 27 | +every persist asserts the stored offset is not greater than the one being written, so the |
| 28 | +newest-by-offset write wins, whoever issued it. The write is linearizable per partition key, so |
| 29 | +concurrent writers to one key are ordered without relying on clock synchronisation: |
| 30 | + |
| 31 | +```sql |
| 32 | +UPDATE snapshots_v2 SET ... , offset = :offset WHERE <key> IF offset <= :offset |
| 33 | +``` |
| 34 | + |
| 35 | +A key's first write finds no row, so the `UPDATE` does not apply and it falls back to |
| 36 | +`INSERT ... IF NOT EXISTS` (retried once via the `UPDATE` if it loses an insert race). This is the one |
| 37 | +non-atomic path — a compound of separate Paxos transactions — but it is safe by construction: both |
| 38 | +`UPDATE`s are offset-gated and the `INSERT` only writes an absent cell, so no interleaving overwrites a |
| 39 | +newer snapshot. A rejected write raises `SnapshotWriteConflict` — including a spurious one if a delete |
| 40 | +slips between the first-write `INSERT` and its retry (an over-rejection, never corruption, cleared on |
| 41 | +the next flush). |
| 42 | + |
| 43 | +The guard is per **key**, the right granularity: #732 corruption is per key and keys are independent, |
| 44 | +so per-key monotonic durability is exactly what prevents it. |
| 45 | + |
| 46 | +## Scope: persist-only |
| 47 | + |
| 48 | +The shipped mode is exactly the mechanism above and nothing more: **`persist` is offset-gated; `delete` is |
| 49 | +untouched** — plain last-write-wins, no new read, no buffer change. So during a rebalance overlap a stale |
| 50 | +writer can still erase a newer snapshot, or resurrect a just-deleted key by writing at a lower offset — |
| 51 | +#732 for that key. It is a bounded scope, not a correctness hole: a deletion that must be fenced is still |
| 52 | +expressible without a gated `delete`, by persisting an offset-carrying *empty* state through the same |
| 53 | +fenced path, so a lower-offset zombie write is rejected exactly as for any persist. |
| 54 | + |
| 55 | +Gating `delete` itself is deferred: it would need a source/binary-breaking `delete(key, offset)` API and |
| 56 | +recovery machinery — a monotonic buffer with a floor seeded from the tombstone — to avoid a replay-window |
| 57 | +livelock. That design is in *Conditional deletes* below. Persist-only is free of the livelock: |
| 58 | +`SnapshotFold`'s existing offset dedup keeps a persist from ever re-deriving below its recovered |
| 59 | +high-water, so the floor comes for free. |
| 60 | + |
| 61 | +## Equal-offset writes and determinism |
| 62 | + |
| 63 | +`IF offset <= :offset` admits an *equal* offset, deliberately: the legitimate owner can write at an offset |
| 64 | +it has already stored — a timer-driven re-flush of the buffered high-water snapshot — and a strict `<` |
| 65 | +would reject that and fence the owner against itself. Admitting equal is safe not because the value is |
| 66 | +identical but because a same-offset write does not move the recovery point — unlike a lower-offset write, |
| 67 | +it cannot drop committed events (#732). |
| 68 | +Any two snapshots at the same offset fold the same records, differing at most in time-driven tick state, |
| 69 | +so deterministic, replayable folds are a precondition of the mode (as they already are of recovery |
| 70 | +generally). |
| 71 | + |
| 72 | +## Rejected alternatives |
| 73 | + |
| 74 | +- **Offset-as-write-timestamp (LWW register)**: write each snapshot `USING TIMESTAMP <offset>` and let |
| 75 | + Cassandra's last-write-wins reconciliation keep the highest-offset cell — a plain quorum write, much |
| 76 | + cheaper than a Paxos round, and a delete becomes a tombstone ordered by offset. Rejected as the |
| 77 | + default: equal-offset replacement breaks (at equal timestamps Cassandra breaks ties by value, not |
| 78 | + write order), a rolling deploy inverts catastrophically (old instances write wall-clock timestamps |
| 79 | + that dominate every offset-as-timestamp value), and it discards the real write timestamps. |
| 80 | +- **Lease / ownership table**: a per-partition lease acquired with one LWT, then cheap writes. The |
| 81 | + lease alone does not stop a paused leaseholder's plain writes (last-write-wins still applies), so a |
| 82 | + per-write fencing token is still required — at which point the lease only adds liveness/expiry |
| 83 | + concerns on top of the per-write CAS. |
| 84 | +- **Composite `(offset, generation)` token**: gate on the consumer generation as well as the offset, |
| 85 | + closing the equal-offset gap and giving per-partition (not just per-key) ownership. Couples the |
| 86 | + self-contained Cassandra module to the live consumer generation; reasonable as a future strict mode, |
| 87 | + not a default. |
| 88 | +- **Recovery-side reconciliation** (store the offset, recover from the lowest): does not prevent the |
| 89 | + stale overwrite (last-write-wins still corrupts), so strictly weaker than fencing the write. |
| 90 | + |
| 91 | +## Forward-looking |
| 92 | + |
| 93 | +- **Offset-gated deletes** — make `delete` safe by default, without the empty-state workaround (see |
| 94 | + *Scope*); the prototyped, TLA+-modelled design is in *Conditional deletes: the deferred design* below. |
| 95 | + Deferred for the source/binary-breaking `delete(key, offset)` API and the recovery machinery — a |
| 96 | + candidate for a future major version. |
| 97 | +- **Per-partition ownership** — the equal-offset gap and per-key (rather than per-partition) granularity |
| 98 | + could be closed by a composite `(offset, generation)` token (see Rejected alternatives) or, further out, by |
| 99 | + [KIP-939 (participation in 2PC)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-939:+Support+Participation+in+2PC): |
| 100 | + a transactional producer in an externally-coordinated two-phase commit could bind the Cassandra |
| 101 | + snapshot write to a generation-fenced Kafka input-offset commit, giving Cassandra per-partition |
| 102 | + ownership without the per-key compare-and-set. Not actionable now; see the Kafka design doc's |
| 103 | + forward-looking note. |
| 104 | + |
| 105 | +## Conditional deletes: the deferred design |
| 106 | + |
| 107 | +**None of this is in the shipped mode.** Fencing deletes was designed and TLA+-modelled, then deferred in |
| 108 | +favour of persist-only; everything below — the tombstone, the `recover` read, the buffer floor, the |
| 109 | +recovery changes — is that deferred design. It follows from a single goal — fence a delete the way a |
| 110 | +persist is fenced, on its offset — with each part forced by the one before it: |
| 111 | + |
| 112 | +**A delete cannot remove the row.** Removing the row removes the `offset` guard with it, so a lagging |
| 113 | +zombie's `INSERT ... IF NOT EXISTS` at a lower offset would then succeed and resurrect a stale snapshot — |
| 114 | +#732 for that key. A fenced delete is therefore a logical **tombstone** — the row kept with its `offset`, |
| 115 | +value nulled, on the same offset-gated path as a persist: |
| 116 | + |
| 117 | +```sql |
| 118 | +UPDATE snapshots_v2 SET value = null, offset = :offset WHERE <key> IF offset <= :offset |
| 119 | +``` |
| 120 | + |
| 121 | +A lower-offset writer is rejected, not resurrected; a replayed delete is a no-op (equal offset) or a |
| 122 | +conflict, never a revival. Keeping the row also holds the delete on the Paxos path, avoiding the hazard of |
| 123 | +mixing lightweight transactions and plain mutations on one row. |
| 124 | + |
| 125 | +**A monotonic buffer, or the owner fences itself.** In the replay window a key can be recovered at its |
| 126 | +durable offset `X` while the partition resumes from a lower committed offset `C` (a slow key held `C` |
| 127 | +back). If the buffer regressed to a replayed offset `< X`, a tick-delete or flush would write below `X`, |
| 128 | +`IF offset <= X` would reject it, and the fence would crash the *legitimate* owner. So the **per-key** |
| 129 | +buffer is kept monotonic in offset — a lower-offset write dropped, a tombstone lifted to |
| 130 | +`max(write offset, that key's high-water)` — and a delete is gated on that key's high-water `X`, which |
| 131 | +the true owner presents and a genuinely stale writer (which only ever reached its own lower offset) does |
| 132 | +not. This is sound under the determinism the design already assumes: re-folding events `<= X` reproduces |
| 133 | +the same state. |
| 134 | + |
| 135 | +**Re-seed the floor on recovery — on both paths, and the second is easy to miss.** The base read is |
| 136 | +`get`, typed `Option[S]`: it returns *a value* or *nothing*, never *deleted at offset `X`*. A tombstone |
| 137 | +therefore comes back through `get` as a plain `None`, indistinguishable from a never-written key, so |
| 138 | +recovery sets no floor — the buffer climbs from the replayed offsets `< X`, the offset-`X` tombstone |
| 139 | +rejects the owner, and the flow livelocks (tear down → re-recover the floorless tombstone → repeat). The |
| 140 | +fix is a *new* recovery read that surfaces the offset — `recover`, returning `Deleted(offset)` where `get` |
| 141 | +returns `None` — to seed the floor. |
| 142 | + |
| 143 | +That repairs **snapshot** recovery. **Events-recovery** re-opens the same livelock on its own: it rebuilds |
| 144 | +state by folding the *journal*, not by reading the snapshot store, and a delete *clears the journal*, so |
| 145 | +the fold yields nothing and the floor is lost again — a path the snapshot-recovery fix never touches. It |
| 146 | +must seed the floor from the snapshot store separately (a read for its side-effect) before folding. **A |
| 147 | +delete fence that seeds the floor only on snapshot recovery still livelocks on events-recovery** — the two |
| 148 | +seedings are independent, and the events-recovery one fires only for a deleted key (a live key's journal |
| 149 | +reconstructs `X` itself). |
| 150 | + |
| 151 | +**What the model proved.** The livelock was modelled in TLA+ as a checked negative control: with the |
| 152 | +monotone buffer or the tombstone floor removed, liveness fails while safety still holds (a rejected write |
| 153 | +changes nothing; `X` never regresses) — a pure liveness failure, reached through a live snapshot in one |
| 154 | +configuration and through a deleted key in another. With both in place the specification refines an |
| 155 | +abstract single-writer store for safety **and** liveness. The deleted-key case needs its *own* floor fix |
| 156 | +precisely because the tombstone reads back as absent — which is also why persist-only is livelock-free: |
| 157 | +re-deriving below the high-water is the everyday persist case `SnapshotFold` already covers. |
0 commit comments