Use Cases for GKE Block SSDs

[nick-stroud]

Spin off discussion from #1506.

When acceptable, it seems much simpler for users to use the ephemeral SSDs than the block SSDs as they do not have to worry about formatting and RAIDing. For the sake of providing a simpler user experience, we had considered holding off exposing the block config unless there is a clear case for it.

Is there a specific use case where block SSDs would be useful?

Interested in feedback from anyone but specifically hoping that @issacg might chime in.

For reference by ephemeral SSDs and block SSDs I am referring to ephemeral_storage_local_ssd_config and local_nvme_ssd_block_config respectively from the google_container_cluster.

[issacg]

First, I agree, and plan on documenting that the recommended best-practice is to use ephemeral_storage_local_ssd_config and emptyDir wherever possible. I intend to make a section to mention this above the terradoc auto-generated stuff.

Secondly, my thoughts were to make local_nvme_ssd_block_config available (after making ephemeral_storage_local_ssd_config available) without guessing the exact use cases because HPC implies high IOPS, and that means that even as an opinionated framework, this seems like it should be on our radar (moreso: if it didn't cost money, it ought to be a sane-default). And since we shouldn't be locking users into using emptyDir, we need to provide other (flexible) access to this storage space. As an OSS project, IMHO although it's important to KISS, that doesn't mean to withhold such an important tuning variable making it impossible for users to accomplish this without forking the project just because we don't have a clear case / request up-front.

Having said that, some ideas on use-case directions:

1. Allowing hyperparameter optimization jobs running in parallel pods to fetch the data locally and then share it with other pods which are then scheduled onto the same node. Then the hyperparameter tuning jobs could use the same local storage shared as a PV in ReadOnlyMany mode, rather than each pod fetching and duplicating the data (bandwidth and/or diskspace constraints).
2. MPI jobs (yes, I'm being optimistic about the future of ultra-high-speed inter-node networking in GCP + GKE, but I leave that to your HPC solution team hats) which want to use PVs to further enhance/optimize intra-node pods in addition to an NFS or other remote storage.
3. Usage of a third-party job scheduler that doesn't support emptyDir (this was almost my real use-case, but emptyDir happened to be on their roadmap due to release in a month).

[cboneti]

folks, this discussion has been stale since June 28, I will close this discussion for now. Please feel free to reopen it if new data arises. :)