For #1449, preview is being used to work with DataFrames pre-loaded by a Curator stage. We can look into switching over to create that generates durable artifacts on disk.
Context: preview is not meant for any meaningful larger generation. While it will let you specify a large number of rows to generate, it doesn't have the durability mechanisms built into create that lets you resume from checkpoints and scale across concurrent row groups. preview also exposes your workers vulnerable to OOMs, at the mercy of the size of each partition a Curator stage loads and the underlying Data Designer pipeline.
Thread: #1449 (comment)
For #1449,
previewis being used to work with DataFrames pre-loaded by a Curator stage. We can look into switching over tocreatethat generates durable artifacts on disk.Context:
previewis not meant for any meaningful larger generation. While it will let you specify a large number of rows to generate, it doesn't have the durability mechanisms built intocreatethat lets you resume from checkpoints and scale across concurrent row groups.previewalso exposes your workers vulnerable to OOMs, at the mercy of the size of each partition a Curator stage loads and the underlying Data Designer pipeline.Thread: #1449 (comment)