Are there tutorials using Distributed Training Libraries and Multibackend?

[santi-mir]

Hi!

I've been tinkering with keras tuner for learning purposes, but doing some searches I've found 2 libraries don't use gRPC.

One of them is Horovod, the other one is Ray, both are open source projects.

I wonder if you have tried any of these, or have any tutorials or thoughts on distributed training? I have used HPC setups with PBS but that's a different level of complexity.

Maybe you prefer gRPC for a specific reason?

[Kxrma47]

Multi-backend Keras makes the model/layer code portable; it does not make the distributed runtime identical across backends. Distribution is still owned by the selected backend:

• TensorFlow uses tf.distribute strategies and its worker coordination.
• JAX uses device meshes/sharding; Keras' Distribution API is built around that model.
• PyTorch uses its native distributed stack, typically DDP.

Horovod and Ray solve different parts of the system. Horovod supplies distributed gradient collectives through MPI/NCCL/Gloo. Ray primarily provides process/cluster orchestration and libraries such as Ray Train/Tune; it can launch backend-specific training workers. Whether either works cleanly depends on the chosen Keras backend and the integration version.

So gRPC is not a reason to choose or reject either library—it is an implementation detail of some coordination paths, not the Keras model API.

For a first setup, I would choose one backend and its native distribution strategy, make a single training job reproducible, then use Ray only if you need cluster scheduling or parallel hyperparameter trials. Trying to make one distributed script transparently interchangeable among TensorFlow, JAX, and PyTorch usually exposes backend-specific checkpointing, input-pipeline, collective, and fault-tolerance behavior even when the model code itself is portable.