Uploading Splits to Hugging Face

[mannuch]

Hello!

Does the hugging face integration support uploading different splits (train/validation/test)? I see that the integration enables processor outputs to get their own top-level directories on HF, but there doesn't seem to be a way to express splits.

Thanks!

[darekwojciechowski]

Not supported today. upload_dataset() pushes the whole parquet-files/ folder in one upload_folder() call with path_in_repo="data" hardcoded in _upload_main_dataset_files, and nothing along that path takes a split argument. The top-level directories you spotted come from _upload_processor_files, which names them after the processor directory, so those are processor outputs rather than splits.

If you need splits on the Hub, build the DatasetDict yourself and skip the built-in uploader:

from datasets import load_dataset

ds = load_dataset("parquet", data_files={
    "train": "train-run/parquet-files/*.parquet",
    "validation": "val-run/parquet-files/*.parquet",
})
ds.push_to_hub("your-user/your-dataset")

You lose the generated dataset card and the metadata.json upload that way, so it's a tradeoff rather than a drop-in replacement.

Quickest check afterwards is the Dataset Viewer on your repo page. A split dropdown above the table means the Hub picked them up.