We're seeing external sort failures where Sedona reports spillable memory is still available, but the allocation is rejected anyway.
The root cause appears to be that SedonaFairSpillPool splits the spill budget across all registered spill-capable consumers, including consumers that are idle and have reserved 0 bytes.
Minimal test case:
#[test]
fn test_idle_spillers_do_not_split_budget() {
let pool: Arc<dyn MemoryPool> = Arc::new(SedonaFairSpillPool::new(100, 0.0));
let active_consumer = MemoryConsumer::new(active).with_can_spill(true);
let mut active = active_consumer.register(&pool);
let _idle_reservations: Vec<_> = (0..128)
.map(|i| {
MemoryConsumer::new(format!(idle-{i}))
.with_can_spill(true)
.register(&pool)
})
.collect();
active.try_grow(100).unwrap();
}
On current main, this fails with:
Failed to allocate additional 100 bytes for active with 0 bytes already allocated - maximum available is 0 bytes.
Current unspillable memory usage: 0 bytes, spillable memory available: 100 bytes
Expected behavior: idle spill-capable consumers should not reduce the memory available to the active consumer.
Observed behavior: the active consumer effectively gets a 0-byte limit because the 100-byte pool is divided across 129 registered consumers, even though only one consumer is using memory.
This seems related to sort-heavy queries with many ExternalSorter consumers, where some sorters are idle but still reduce the budget for active sorters.
We're seeing external sort failures where Sedona reports spillable memory is still available, but the allocation is rejected anyway.
The root cause appears to be that
SedonaFairSpillPoolsplits the spill budget across all registered spill-capable consumers, including consumers that are idle and have reserved 0 bytes.Minimal test case:
On current
main, this fails with:Expected behavior: idle spill-capable consumers should not reduce the memory available to the active consumer.
Observed behavior: the active consumer effectively gets a 0-byte limit because the 100-byte pool is divided across 129 registered consumers, even though only one consumer is using memory.
This seems related to sort-heavy queries with many
ExternalSorterconsumers, where some sorters are idle but still reduce the budget for active sorters.