Summary
Several bugs in the Lotus sealing pipeline cause sectors to get stuck in perpetual retry loops, block callers indefinitely, and perform non-deterministic allocation validation. This issue tracks fixes across commit_batch.go, input.go, and precommit_policy.go.
Issues Fixed
1. Sectors stuck in infinite retry loop (commit_batch.go)
Sectors experiencing non-retryable failures (e.g., missing precommit info, collateral calculation errors) are retried indefinitely with no eviction mechanism. This causes resources to be wasted on sectors that will never succeed.
Fix: Track consecutive non-retryable failures per sector and evict after 5 consecutive failures. Evicted sectors receive a failure result delivered to all waiters, and their state is cleaned up.
2. AddCommit callers blocked forever when sector is silently dropped (commit_batch.go)
If a sector is queued for batch processing but processBatchV2 drops it without producing a result (e.g., when infos is empty and res.Sectors is empty), the caller in AddCommit blocks indefinitely waiting for a result that never arrives.
Fix: Added a defensive fallback after processBatchV2 that iterates over all sectors in the batch and ensures every sector receives a result. Sectors missing from the response get an explicit failure result. Also fixed processBatchV2 to return res when infos is empty but res.Sectors has failures.
3. Non-deterministic allocation TermMin/TermMax check (commit_batch.go)
The allocationCheck method uses live ts.Height() as the reference point for TermMin/TermMax validation. Since chain height advances between retries, an allocation can flip from valid to invalid (or vice versa) non-deterministically, causing inconsistent behavior.
Fix: Changed to use PreCommitEpoch as a fixed reference point, making validation deterministic.
4. Missing allocation term validation (input.go)
getClaimTerms did not validate that an allocation's TermMax is sufficient for the sector's minimum lifetime requirement. This could allow sectors to be created with allocations that expire before the sector's required minimum expiration.
Fix: Added minAllocationTermForSealProof which computes the minimum allocation term based on the seal proof's max prove commit duration, precommit randomness lookback, and minimum sector expiration. getClaimTerms now rejects allocations with TermMax below this minimum.
5. Missing allocation validation in precommit policy (precommit_policy.go)
BasicPreCommitPolicy.Expiration did not validate piece CID and size against the on-chain allocation, and did not clamp the deal end epoch to the allocation's TermMax. This could lead to sectors being precommitted with expiration beyond what the allocation allows.
Fix: Added allocation lookup and validation:
- Verify piece CID and size match the on-chain allocation
- Clamp
endEpoch to TermMax if the deal end epoch exceeds it
Affected Files
storage/pipeline/commit_batch.go
storage/pipeline/input.go
storage/pipeline/precommit_policy.go
Summary
Several bugs in the Lotus sealing pipeline cause sectors to get stuck in perpetual retry loops, block callers indefinitely, and perform non-deterministic allocation validation. This issue tracks fixes across
commit_batch.go,input.go, andprecommit_policy.go.Issues Fixed
1. Sectors stuck in infinite retry loop (
commit_batch.go)Sectors experiencing non-retryable failures (e.g., missing precommit info, collateral calculation errors) are retried indefinitely with no eviction mechanism. This causes resources to be wasted on sectors that will never succeed.
Fix: Track consecutive non-retryable failures per sector and evict after 5 consecutive failures. Evicted sectors receive a failure result delivered to all waiters, and their state is cleaned up.
2. AddCommit callers blocked forever when sector is silently dropped (
commit_batch.go)If a sector is queued for batch processing but
processBatchV2drops it without producing a result (e.g., wheninfosis empty andres.Sectorsis empty), the caller inAddCommitblocks indefinitely waiting for a result that never arrives.Fix: Added a defensive fallback after
processBatchV2that iterates over all sectors in the batch and ensures every sector receives a result. Sectors missing from the response get an explicit failure result. Also fixedprocessBatchV2to returnreswheninfosis empty butres.Sectorshas failures.3. Non-deterministic allocation TermMin/TermMax check (
commit_batch.go)The
allocationCheckmethod uses livets.Height()as the reference point for TermMin/TermMax validation. Since chain height advances between retries, an allocation can flip from valid to invalid (or vice versa) non-deterministically, causing inconsistent behavior.Fix: Changed to use
PreCommitEpochas a fixed reference point, making validation deterministic.4. Missing allocation term validation (
input.go)getClaimTermsdid not validate that an allocation'sTermMaxis sufficient for the sector's minimum lifetime requirement. This could allow sectors to be created with allocations that expire before the sector's required minimum expiration.Fix: Added
minAllocationTermForSealProofwhich computes the minimum allocation term based on the seal proof's max prove commit duration, precommit randomness lookback, and minimum sector expiration.getClaimTermsnow rejects allocations withTermMaxbelow this minimum.5. Missing allocation validation in precommit policy (
precommit_policy.go)BasicPreCommitPolicy.Expirationdid not validate piece CID and size against the on-chain allocation, and did not clamp the deal end epoch to the allocation'sTermMax. This could lead to sectors being precommitted with expiration beyond what the allocation allows.Fix: Added allocation lookup and validation:
endEpochtoTermMaxif the deal end epoch exceeds itAffected Files
storage/pipeline/commit_batch.gostorage/pipeline/input.gostorage/pipeline/precommit_policy.go