Skip to content

Commit ed53c6d

Browse files
authored
Drop ParallelIterable's queue low water mark (apache#10979)
As part of the change in commit 7831a8d, queue low water mark was introduced. However, it resulted in increased number of manifests being read when planning LIMIT queries in Trino Iceberg connector. To avoid increased I/O, back out the change for now.
1 parent e18a2fe commit ed53c6d

1 file changed

Lines changed: 3 additions & 6 deletions

File tree

core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -192,12 +192,9 @@ public synchronized boolean hasNext() {
192192
// If the consumer is processing records more slowly than the producers, the producers will
193193
// eventually fill the queue and yield, returning continuations. Continuations and new tasks
194194
// are started by checkTasks(). The check here prevents us from restarting continuations or
195-
// starting new tasks too early (when queue is almost full) or too late (when queue is already
196-
// emptied). Restarting too early would lead to tasks yielding very quickly (CPU waste on
197-
// scheduling). Restarting too late would mean the consumer may need to wait for the tasks
198-
// to produce new items. A consumer slower than producers shouldn't need to wait.
199-
int queueLowWaterMark = maxQueueSize / 2;
200-
if (queue.size() > queueLowWaterMark) {
195+
// starting new tasks before the queue is emptied. Restarting too early would lead to tasks
196+
// yielding very quickly (CPU waste on scheduling).
197+
if (!queue.isEmpty()) {
201198
return true;
202199
}
203200

0 commit comments

Comments
 (0)