11_big_data

11 — Big data with IterableDataset

The Dataset from lesson 07 assumes you can ask "give me the i-th sample" in O(1). That's fine for a numpy array, but breaks down when:

• Your data lives in a 500 GB CSV.
• It's streamed from Kafka / a network socket.
• It's spread across thousands of shards on S3 / GCS.

Use IterableDataset instead. You implement __iter__ and yield samples one at a time. PyTorch handles the rest.

Pattern

class MyStream(IterableDataset):
    def __iter__(self):
        for record in some_streaming_source():
            yield preprocess(record)

Memory stays constant in the dataset size — perfect for "bigger than RAM".

Multi-worker sharding

When DataLoader(num_workers > 1), each worker runs __iter__ independently. Without sharding logic, every worker reads the full file and you get duplicate samples. Inside __iter__:

info = torch.utils.data.get_worker_info()
if info is not None:
    # shard by worker id, e.g. only read every (info.num_workers)-th line
    ...

Production big-data stack (the real world)

For TB-scale ML, people use specialized formats and frameworks:

• Apache Arrow / Parquet — columnar, mmap-friendly, zero-copy.
• WebDataset / TFRecord — sharded tar files of pre-processed examples; perfect for IterableDataset.
• HuggingFace datasets — Arrow-backed, supports streaming from the hub.
• Spark / Ray Data — distributed preprocessing into shards your loader consumes.

The pattern is always the same: chunk the data into shards, stream them, do preprocessing in the loader workers, hand batches to the GPU. This lesson shows the smallest version of that pipeline.

Run

python 11_big_data/main.py