Question: Manual Dataset Sharding per GPU Rank with Accelerate + DistributedSampler (Avoid Double DataLoader Length Split)

[D222097]

Hi, I'm training on a very large dataset with LeRobot (which uses Hugging Face Accelerate) across multiple GPUs.

To minimize cross-process memory usage, I want to manually shard the full dataset by GPU rank BEFORE creating the DataLoader (following this approach: https://github.com/huggingface/datasets/issues/8217).

However, I’m running into the issue where the DataLoader length gets split twice — once by my manual DistributedSampler and again by Accelerate (tracked here: https://github.com/huggingface/accelerate/issues/3520).

My Goal:

• Shard the full large dataset per GPU rank upfront (before DataLoader initialization) to reduce per-process memory footprint
• Prevent Accelerate from automatically splitting the DataLoader/dataset a second time
• Maintain correct multi-GPU training behavior

Questions:

• What’s the correct way to manually shard a dataset by GPU rank when using Accelerate?
• How can I disable Accelerate’s automatic DataLoader splitting to avoid the double-shard issue?
• Are there specific settings (e.g., Accelerator kwargs, DataLoader/DistributedSampler flags) required for this workflow?

Any suggestions would be greatly appreciated!

[D222097]

Does this double split (DistributedSampler + Accelerator.prepare(dataloader)) affect training? I just want to know if it will reduce the training data size or cause incorrect training.

Suppose I have a dataset with total x samples, training on 2 GPUs:
I manually use DistributedSampler, so each GPU gets x/2 samples initially.
Then I call accelerator.prepare(dataloader).

What does Accelerate actually do here?
Does it keep each GPU at x/2 (no problem)?
Or does it split again, so each GPU ends up with only x/4 samples?
Or does it shuffle data between GPUs?

I’m worried this will reduce the effective training data size or break training correctness.
Does anyone know the actual behavior? Thanks a lot!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.