[Feature] Unsloth studio: Set dataset streaming mode

[fablevi]

Can you add this somehow to the UI?

raw_dataset = load_dataset("facebook/voxpopuli", "en", split="train", streaming=True)

[Datta0]

We'd appreciate if someone from the community can contribute

[sanatb187]

I'd love to work on this issue! Could you please assign it to me?

[sanatb187]

Hi @Datta0, just wanted to clarify the expected scope here: should this streaming mode be limited to Hugging Face datasets, or do you want true end-to-end training support for IterableDatasets across HF and local datasets?

[Datta0]

ig thats a question OP has to answer? But we can take things step by step if that makes sense

[fablevi]

Hey! @sanatb187

I think probably in long term the local and HF support would be good.

But for now, i tought about only HF.

Thanks!

[sanatb187]

Makes sense. We could scope this issue to just enable streaming behavior for HF datasets, and create a subsequent issue to add end to end support for local datasets. Happy to contribute to that one too!

[sanatb187]

Hi @Datta0 I opened a PR for the HF dataset streaming part of the issue. I validated it on GPU with streamed HF training runs (with and without eval enabled). At this point, you wouldn't see the streaming option for local datasets, but we could work on adding support for streaming of local datasets in a subsequent PR.