When mapping a callback to the zcollection partitions, we might want to specify the 'resources' parameter when submitting the tasks.
zcollection.Collection.update:
batches: Iterator[Sequence[str]] = dask_utils.split_sequence(
selected_partitions, npartitions
or dask_utils.dask_workers(client, cores_only=True))
storage.execute_transaction(
client, collection.synchronizer,
# client.map(local_func, batches, resources=resources, key=func.__name__)) CURRENT CODE
client.map(local_func, batches, resources=resources, key=func.__name__)) # REQUESTED CHANGE
In case we parallelize a long running task, dask worker may declare itself as idle if the blocking task releases the GIL or is blocked doing IO. Setting resources can help us limit overscheduling due to this behavior.
When mapping a callback to the zcollection partitions, we might want to specify the 'resources' parameter when submitting the tasks.
zcollection.Collection.update:
In case we parallelize a long running task, dask worker may declare itself as idle if the blocking task releases the GIL or is blocked doing IO. Setting resources can help us limit overscheduling due to this behavior.