Skip to content

pdp: eth sender flattens EIP-1559 transactions to legacy at nonce assignment - every message pays its full fee cap and strands on any baseFee rise #1354

Description

@TippyFlitsUK

Checklist

Curio component

  • Other (eth message sender - tasks/message/sender_eth.go)

Curio Version

curio version 1.28.2-rc2+mainnet+git_c98f048c_2026-06-12T00:34:03+00:00

All three defects are also present in tasks/message/sender_eth.go on current main (blob 4f49253).

Describe the Bug

To share the load and speed up the Storacha migration I spun up a second mainnet PDP node alongside our existing one. It initially did more harm than good - between them the two nodes generated enough gas demand to drive the baseFee up against their own messages, and both started stranding. Profiling why led us to three compounding defects in the eth sender:

  1. EIP-1559 transactions are silently downgraded to legacy at nonce assignment. Send() builds a correct DynamicFeeTx (cap + tip), but the nonce-assignment step rebuilds it with types.NewTransaction(assignedNonce, ..., tx.GasPrice(), ...) - the legacy constructor. geth's GasPrice() returns the fee cap for a dynamic-fee tx, so the cap/tip split is collapsed into a single legacy gasPrice equal to the cap. Every PDP message on chain is type: 0x0.

  2. The sender always pays its full cap. Lotus maps legacy txs as GasFeeCap = GasPremium = gasPrice, so effective premium = cap minus baseFee, charged on the full gas limit. Six sampled receipts show effectiveGasPrice 4-7% above the tx gasPrice (premium on unused gas limit plus overestimation burn). A baseFee dip does not save the SP money - it increases the miner tip.

  3. Zero headroom, no recovery. The cap is priced at baseFee + gasTipCap at send time, so any baseFee rise strands the message, and there is no fee-bumping path (Curio should be able to replace message when there is a baseFee spike #1292). Nonces are sequential, so one stranded message blocks the SP wallet's entire lane - including PDP proving messages (pdp: separate sender lane so dealbot/proving messages don't queue behind bulk addPieces #1313 is the same shared-lane design).

What this did to us in practice (profiled 2026-07-10, both SPs, 30s-resolution mempool and baseFee data, per-tipset gas attribution):

  • Our two nodes alone were 85-86% of all tipset gas over an 80-tipset window - a bulk workload self-generates the baseFee ramps that strand its own messages. Stopping both nodes collapsed baseFee 960k to 505k in ~7 minutes; restarting ramped it straight back.
  • Stock behaviour: one lane pinned for 2 1/4 hours behind a single stale-cap head (918,479 cap vs baseFee oscillating 825k-1.14M) with nineteen includable messages (caps to 2.19M) nonce-blocked behind it. Client timeouts cycle fresh sends in behind the stuck head - we accumulated 308 stranded messages overnight.
  • Proving hostage, twice in one day. Proving messages share the wallet lane. A stranded migration message left data set 1342 39 minutes from missing its proving window (recovered only by manually halting the other SP to collapse baseFee). Later, with mitigations running but deep client queues, a second 30+ minute pin formed minutes after data sets 1244, 1255 and 99 got their proofs out - had the pin formed ten minutes earlier, all three would have missed their windows. An unattended SP in this state has a fault exposure, not a throughput problem.

Expected behaviour: 1559 semantics preserved end to end - the cap is a ceiling, the tip is the price signal, a message survives ordinary baseFee movement, and inclusion at a lower baseFee costs less, not more.

Mitigations we tested (running on both SPs, happy to PR):

  1. Preserve the DynamicFeeTx through nonce assignment (type-conditional - legacy constructions elsewhere untouched).
  2. gasFeeCap = 2 x baseFee + gasTipCap.
  3. Floor the baseFee input at the max of eth_feeHistory over the last 120 epochs, so a restart into a quiet chain does not price messages against a transient floor (this exact trap stranded our first patched run within 15 minutes - caps born at a 24k floor, dead by 300k).

Measured results: messages now pay baseFee + tip instead of full cap; a restart from baseFee 183 (the harshest bootstrap case - first caps ~2.39M off the history floor) ran ~2 hours with both lanes confirming continuously through a 2,000x self-ramp, where the unfloored run pinned within 15 minutes.

The limit we then hit, and the reason pricing alone is not the fix: under sustained dual full-rate load (16 in-flight messages per node), messages queue long enough that even 2x caps go stale inside the self-generated ramp - one node pinned 30+ minutes at a 435k head against an 855k baseFee while the other node's fresh caps kept clearing. We are re-running at halved client concurrency so injection matches the chain's clearing rate, which should remove the self-induced cap staleness - but the correctness fix is #1292: same-nonce replacement for underwater messages, with confirmation tracked by nonce so either hash resolves. Our data above is effectively the severity case for it.

Logging Information

eth_getTransactionByHash (any AddPieces tx, both SPs):  type: 0x0, gasPrice only, no maxFeePerGas
receipts (6 consecutive confirmed AddPieces):           cap 953932 -> paid 990720 | cap 953006 -> paid 970450
                                                        cap 959064 -> paid 1019407 | cap 965199 -> paid 1023162
                                                        cap 968439 -> paid 1026390 | cap 973185 -> paid 1031710
stalled lane snapshot (nonce-ordered pending):          head cap 918479 vs baseFee 825k-1.14M; caps interleaved
                                                        852126-2186904 behind it; all GasPremium == GasFeeCap

Full 30s-resolution TSVs from both nodes plus per-tipset gas attribution available on request.

Repo Steps

  1. Run any PDP node with addPieces load through SenderETH.
  2. eth_getTransactionByHash any resulting tx: type: 0x0, no maxFeePerGas.
  3. Compare effectiveGasPrice in the receipt to the tx gasPrice: paid >= cap, in all baseFee conditions.
  4. Sustain enough load to move baseFee upward past send-time caps: the lane strands, including proving messages, with no recovery until baseFee dips below the head's cap.

Metadata

Metadata

Assignees

No one assigned

    Labels

    team/fs-wgItems being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    🐱 Todo

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions