Devnet 1.3.0 Frontier Smoke Report

[rvagg]

What follows is heavily AI assisted; my devnet testing was AI orchestrated and this report is condensed from the findings by the AI with editing by me..

This smoke ran an accelerated local FOC stack to exercise the storage lifecycle across Synapse, Curio, FWSS, PDPVerifier, and FilecoinPay. The goal was not to benchmark devnet, but to walk the release-critical paths quickly enough to observe creation, charging, proving, termination, and cleanup behaviour end to end.

What Was Exercised

The run covered the main user and provider lifecycle:

• Client funding, service approval, price-list reads, dataset creation, and initial piece upload through Synapse.
• Batched upload with 40 pieces, including per-piece operation fee math, lifecycle reserve use, lockup updates, and findPiece behaviour under indexing lag.
• Per-operation lockup replenish after repeated adds.
• Piece removal through Synapse to Curio to PDPVerifier, including scheduled removal, deferred cleanup at the next proving period, and the user removal fee.
• Multiple shortened proving periods, including nextProvingPeriod, proof submission, PDP proof fees, and FilecoinPay settlement.
• SP-relayed client termination through Synapse and Curio, including client signature validation, Curio termination queue/status, FWSS termination, terminate fee, zeroed lifecycle reserve, and zeroed fixed lockup.
• Direct on-chain termination as a control path, confirming it does not charge the SP-mediated terminate fee.
• Normal cleanup after termination/finalisation.
• Abandonment cleanup: stop the owning SP, wait through inactivity, then have a third party call deleteDataSet and cleanupPieces to claim the cleanup deposit.
• Negative/security edges: unauthorised termination, malformed relayed termination payloads, third-party deletion before inactivity, cleanup before cleanup mode, and unauthorised/malformed piece-removal scheduling.

What Went As Expected

The core economic and lifecycle paths mostly behaved as intended.

• Synapse read pricing and prepared funding/approval correctly.
• The 40-piece add path worked; the fee calculation matched base + n * perPiece.
• Repeated adds consumed and then replenished lifecycle reserve/lockup as expected.
• Piece removal was scheduled, charged, and later flushed during proving.
• The SP-relayed termination path completed and produced the expected FWSS/FilecoinPay state changes.
• Direct on-chain termination worked as a useful control and did not apply the SP-mediated terminate fee.
• The third-party abandonment flow completed on-chain, including the cleanup deposit payout, after working around the current PDP cleanup gating behavior.
• The cleanup deposit was paid in full to the third-party cleanup caller; the observed wallet delta was lower only because the caller paid gas.

Problems Found

Several issues surfaced, but they fall into different buckets.

Code-under-test findings

• FWSS: Abandonment teardown failed after proving had activated and later periods had faulted. The flaw shape: abandonment first settled the PDP rail to the last resolvable proving boundary, then immediately terminated and tried to finalise through the current/open proving period. FWSS validation could not advance that second settlement, so FilecoinPay reverted with a no-progress settlement error. This is now covered by filecoin-services PR 512, fix(abandonment): unblock finalize settle past open proving period, which fixes the issue without adding contract state.
• PDPVerifier: The code tested required two inactivity windows for permissionless abandonment cleanup. A third party could call deleteDataSet only after the dataset had been inactive for INACTIVITY_WINDOW, but cleanupPieces then started a second INACTIVITY_WINDOW from cleanup-mode entry before non-SP callers could claim the cleanup deposit. With the production constant this is roughly 60 days total. This is not intended; the fix is PDP PR 279, fix(cleanup): anchor cleanupPieces gate to last proving activity.
• Curio: After successful third-party abandonment cleanup, restarting the abandoned SP did not reconcile local Curio state. Curio still had the dataset/piece rows and repeatedly tried proving work against a dataset that was no longer live on-chain.
• Curio: Normal cleanup cleared FWSS/PDP state and Curio PDP rows, but retrieval by PieceCID still returned bytes. This needs product/security clarification: either retrieval after PDP cleanup is allowed until lower-level GC, or Curio should gate retrieval on active PDP references.

foc-devnet/tooling issues likely needing PRs

• Curio runtime containers were missing the FilecoinPay address env var, so cleanup/finality logic could not run until a devnet env mapping issue was patched.
• Observability remains too manual. Diagnosing failures required stitching together Synapse logs, FWSS views, PDP state, FilecoinPay rails, Curio DB tables, and Docker logs. A compact smoke/status view would make future runs much less error-prone.

Smoke setup/process friction

• Curio rejected SP-relayed termination because its capability gate depended on FWSS VERSION() rather than the callable ABI surface. The ABI supported the method, but the version string lagged. The FWSS version has since been bumped so this should not recur from current main.
• Curio cleanup polling was too slow for an accelerated smoke until locally shortened.

Repeatable Shape

An agent repeating this should:

1. Start a fresh local stack from local Synapse, Curio, and filecoin-services sources with shortened proving, lockup, and inactivity windows.
2. Run create/fund/upload first, then one 40-piece add, then enough repeated adds to force lockup replenish.
3. Schedule one piece removal and wait for the next proving period to flush it.
4. Wait through at least two proving periods before termination tests.
5. Run SP-relayed termination, then direct on-chain termination as a control.
6. Verify normal cleanup: FWSS cleared, PDP dataset gone, FilecoinPay rail finalised, Curio PDP rows gone, and retrieval behaviour explicitly checked.
7. Create a separate dataset for abandonment, stop the owning SP, wait past inactivity, run third-party deleteDataSet, then run third-party cleanupPieces. Against unfixed PDP code, a second cleanup-mode inactivity wait is required; with PDP PR 279, cleanup should be immediate after permissionless delete.
8. Run negative permission checks on a fresh live dataset and contract tests for the proven-abandonment edge.
9. Capture not just pass/fail, but any places where state can only be understood by correlating several systems.

Bottom Line

The smoke validated most intended 1.3.0 lifecycle behavior, but it also found several release-relevant follow-ups: the FWSS abandonment settlement flaw for active/faulted datasets, the PDP two-window abandonment cleanup bug, Curio not reconciling externally abandoned datasets after on-chain cleanup, and retrieval still serving bytes after PDP/FWSS cleanup. The FWSS settlement flaw is addressed by PR 512; the PDP cleanup timing bug is addressed by PR 279.

Code Used

This was not a pure main run. It used a deliberately combined frontier stack:

• filecoin-services / FWSS: the pre-merge client-initiated termination flow branch. That branch has since been merged without modification. For the smoke, FWSS/PDP timing constants were shortened locally.
• Synapse: a combined/janky local worktree with pricing/cost calculation and regenerated FWSS ABI/types, SP-relayed terminateService, and the findPiece polling fix.
• Curio: latest local main, with smoke-only patches for devnet disk pressure, relaxed FWSS termination capability detection, and shortened cleanup/finality polling.
• PDPVerifier: local smoke timing patch for a shorter inactivity window, deployed by in-place proxy upgrade. The smoke exposed the unintended two-window cleanup behavior; the fix is PDP PR 279.
• foc-devnet: on top of this PR, with local env fix so Curio received the FilecoinPay contract address at runtime.
• Additional smoke fix found during the run: FWSS abandonment settlement was patched locally after the live abandonment path exposed the FilecoinPay no-progress edge. Abandonment of an already-activated/faulted dataset must not require validated settlement progress through an unresolved proving period. The canonical fix is filecoin-services PR 512.

Acceleration Checklist

The important lesson was that shortening only one layer is not enough. The contracts, Curio task cadence, and devnet runtime env all had to line up.

Effective contract timings used:

• FWSS proving period: 120 epochs.
• FWSS challenge window: 10 epochs.
• PDPVerifier challenge finality: 5 epochs.
• FWSS PDP_INACTIVITY_WINDOW: 300 epochs.
• PDPVerifier INACTIVITY_WINDOW: 300 epochs.
• FWSS pricing lockups: DEFAULT_LOCKUP_PERIOD = 30 epochs and CDN_LOCKUP_PERIOD = 30 epochs.

Notes on those values:

• 300 epochs was short enough to complete abandonment in a reasonable time, but long enough to observe the window and avoid missing it during Curio/SP timing jitter.
• The observed PDP abandonment path had two waits: one before deleteDataSet, then another INACTIVITY_WINDOW after cleanup mode begins before a third party can call cleanupPieces. This is addressed by PDP PR 279. Future accelerated runs should still know about it when testing unfixed code, but fixed code should allow immediate third-party cleanup after permissionless deleteDataSet.
• The lockup periods were intentionally tiny so termination, settlement, and cleanup could finalise during one smoke run.

Curio changes needed for the accelerated run:

• Lowered the piece-store free-space guard for devnet: StorePiece min-free percent 0.
• Reduced maxSizePiece to 4 << 30 to avoid local disk-pressure failures.
• Relaxed the FWSS terminate-service capability/version gate. The method was available, but FWSS VERSION() still reported an older value, so a strict semver gate blocked SP-relayed termination.
• Shortened cleanup/finality task cadence: chain sync from production-scale polling to about 5s, and delete/cleanup task polling from about 10m to about 5s.
• Ensured Curio runtime containers received the FilecoinPay address. This was a foc-devnet env mapping fix, not a protocol change.

Synapse changes needed:

• No timing-specific acceleration patch was needed in Synapse.
• The smoke did require a combined local Synapse worktree with the 1.3.0-facing changes: pricing/cost calculation, regenerated FWSS ABI/types, SP-relayed termination, and the findPiece polling fix.
• For MockUSDFC on devnet, smoke setup needed to avoid assuming permit/nonces support for every funding path.
  • Follow-up: MockUSDFC remains a poor cousin of USDFC, we should work on improving it to be a proper mock.

[rvagg]

As linked above, these are the two main things coming out of this exercise:

• fix(abandonment): unblock finalize settle past open proving period #512
• fix(cleanup): anchor cleanupPieces gate to last proving activity pdp#279

Regarding what was considered "GA" for this:

• Current curio main branch
• Effective filecoin-services main branch and submodules (was this PR now merged)
• Synapse combo of 1, 2 and 3

[BigLep]

Exceptional writeup, @rvagg! The level of detail here, especially the acceleration checklist and repeatable shape sections, makes this genuinely useful as a reference, not just a one-time report. Thank you for the thoroughness.

A few reactions:

On the value of foc-devnet: This is really reinforcing for me how important it is to have foc-devnet for testing our system end-to-end before making breaking changes. It would have saved us in the 2026-05-28 hotfix incident, and it's saving us right now from having to scramble on mainnet fixes that would have happened if we'd rolled out FWSS in its current state.

On complexity: Reading the "What Was Exercised" section is a good example of the intricacy and complexity of the stack. I wish it wasn't that way, but that is the nature of the problem. It drives home why we need mechanisms for validation like this.

Codifying the test scenarios: I'd like to make sure the tests from "What Was Exercised" are codified as foc-devnet scenarios, at minimum as optional ones we can run to confirm we don't regress here in future releases. If there isn't already a tracking issue for this, I'm happy to create one. @rvagg let me know if this is already covered or if a new issue would be useful.

Code-under-test findings: I assume we need Curio items created for the two Curio findings (state reconciliation after external abandonment cleanup, and retrieval still serving bytes after PDP/FWSS cleanup). I'll create those once confirmed, but tagging @ZenGround0, @snadrus, and @LexLuthr so they're aware in the meantime.

foc-devnet/tooling issues: I assume we need backlog items for the items listed under "foc-devnet/tooling issues likely needing PRs" (the FilecoinPay env var mapping and the observability/status view). Will create those as well.

Re-running after fixes land: I think we should run this foc-devnet smoke again once we have updated FWSS (PR #512) and PDP (pdp#279). There are good notes here on what's needed to make it work, but I suspect non-@rvagg people will struggle to execute this end-to-end. I'm open to having others try, but I'm currently thinking the most valuable use of any sliver of your time before international travel, @rvagg, would be a re-run once those fixes are merged.

[BigLep]

Created tracking issues for the items identified above:

Curio (code-under-test findings):

• Curio does not reconcile local state after third-party abandonment cleanup filecoin-project/curio#1283 — Curio does not reconcile local state after third-party abandonment cleanup
• Retrieval still serves bytes after PDP/FWSS cleanup filecoin-project/curio#1284 — Retrieval still serves bytes after PDP/FWSS cleanup

foc-devnet (tooling issues):

• Curio containers missing FilecoinPay address env var foc-devnet#125 — Curio containers missing FilecoinPay address env var
• Add compact smoke/status observability view for cross-system debugging foc-devnet#126 — Add compact smoke/status observability view for cross-system debugging
• Codify 1.3.0 smoke test scenarios as optional foc-devnet scenarios foc-devnet#127 — Codify 1.3.0 smoke test scenarios as optional foc-devnet scenarios