Piece deletion is expensive and non-atomic — consider a bundled replace

[frrist]

Problem

The current piece model forces an uncomfortable trade-off between onboarding cost and deletion cost.

Aggregation is required for any meaningful onboarding

Each piece add costs ~120–300K gas across both contracts: PDPVerifier writes 2 cold SSTOREs in addOnePiece (service_contracts/lib/pdp/src/PDPVerifier.sol:467) plus a sumtree update, and FWSS piecesAdded (service_contracts/src/FilecoinWarmStorageService.sol:841) layers on a nonce SSTORE, ECDSA verify, per-key metadata SSTOREs, and a payment-rate update.

At 100 GB total: ~48M gas as 256 MiB pieces (≈2 txs) vs. ~12B gas as 1 MiB pieces (≈400 txs). Allowing "any data to be a piece" sounds simpler, but is uneconomic for typical workloads. Aggregation isn't an optimization — it's the only way the numbers work.

Aggregation makes deletion painful

Because the piece is the indivisible unit of removal and proving, deleting one user's data out of an aggregated piece means re-aggregating the survivors off-chain and submitting two on-chain messages: schedulePieceDeletions (PDPVerifier.sol:487) for the stale aggregated piece, then addPieces for the replacement.

Each message pays its own base tx gas, ECDSA verify, FWSS nonce SSTORE, and updatePaymentRates external call to the Payments contract — most of which is duplication of what is logically one operation. The two messages are also not atomic: a failure between them leaves the dataset inconsistent and the operator has to recover by hand.

This will get worse as deletion volume grows (compliance-driven deletes, churn-heavy workloads, dataset migrations).

Direction

A bundled replacePieces-style entrypoint that schedules the old piece for removal and adds the new one in a single transaction would address both pains:

• Removes the orphaned-state failure mode entirely.
• Saves the duplicated per-tx overhead — most notably the second updatePaymentRates call, which can be computed once against the net leaf delta.
• Keeps the existing proving-period semantics intact (both halves settle at the next nextProvingPeriod).

The exact shape — listener interface change, signature scheme, batch behavior, fallback for unaware listeners — is open for whoever picks this up.

This issue/analysis was produced using Claude Code.

[Kubuxu]

There's complexity in the deferred execution of the piece addition that I can't directly analyse right now, but it's a good idea overall.
The reason for deferred removal is the proving schedule and the possibility of removing pieces when challenges land on pieces that were not stored.

[rvagg]

To be clear @frrist you're proposing that a replacePieces would involve (internally) what would essentially be a schedulePieceAdditions to pair up with schedulePieceDeletions? Your description isn't entirely clear about the mechanics you want to see but since you reference nextProvingPeriod which is where deletions happen, I'm assuming this is a queued atomic add/remove at that point.

On the non-SP (user, MSP, whatever) side, I think you're saying that the annoyance is atomicity - is this mostly a concern about the possible failure of a deletion msg rather than needing them to be in the same transaction?

On the contract side, there's the cost saving of the rail rate modification, which is good, but we'd have to trade off against some negatives:

1. Additional state in PDPVerifier and probably the service contract to track the scheduled-additions
2. Additional gas cost in nextProvingPeriod to do this work, which I suspect might already be a risk for high-churn data sets (see Measure the gas costs of processScheduledPieceMetadataRemovals for different numbers of deletions #461). We need to try and keep proving costs down, not increase them if we can help it.

Additionally there's a provider revenue gap with this, which could get very large and very annoying: if you're replacing mega pieces then you're asking them to hold on to both old and new until nextProvingPeriod and not getting paid for the new piece while they wait. Currently addPieces bumps the rate as soon as they land the message on chain, which compensates them (a little) for holding both of them during that window. It's not a big gap, but if the primary use case here is very-large-pieces, removing that compensation could matter.

[frrist]

To be clear @frrist you're proposing that a replacePieces would involve (internally) what would essentially be a schedulePieceAdditions to pair up with schedulePieceDeletions?

not a queued atomic add/remove. Immediate add (existing addPieces semantics: rate bumps now, leaves enter challengeRange at next period), queued delete via the existing nextProvingPeriod removal path. No schedulePieceAdditions counterpart. Something like:

  function replacePieces(
      uint256 setId,
      uint256[] calldata oldPieceIds,
      Cids.Cid[] calldata newPieceData,
      bytes calldata extraData
  ) external returns (uint256 firstAdded);

Internally that would call the existing addOnePiece loop for the news + a push to scheduledRemovals for the olds. No new deferred-add execution, no new tracking state in either contract. It's a bundling of the existing flows, not a third execution mode.

I think you're saying that the annoyance is atomicity - is this mostly a concern about the possible failure of a deletion msg rather than needing them to be in the same transaction?

The motivation is gas first, atomicity second (mostly as a side effect). If the bundle menthod can't meaningfully reduce per-op cost, the proposal loses merit.

On the provider revenue gap: immediate-add preserves the existing rate bump on submission, so SPs are compensated as today. The mirror image is on the caller side — they pay for both old and new until the next nextProvingPeriod. But that's already true of today's 2-tx flow. The bundle inherits the overlap; it doesn't widen it. Caller-pays-for-overlap is the existing semantics and this design aims to preserve that.