PLATFORMS Signal 174
We terminated a TPU mid-training and it recovered in seconds: Introduction to elastic training with MaxText
↗
Distributed training jobs typically crash entirely when a single node fails, forcing a full restart from the last checkpoint with significant time and compute costs. This elastic training approach keeps the training process alive and recovers in-place, which could substantially reduce downtime and wasted compute for large-scale model training runs.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Pathways uses a single-controller architecture where one Python process on a CPU machine orchestrates all TPU chips, so when a TPU machine dies the controller remains alive and can handle the failure rather than crashing like SPMD setups.
Recovery was demonstrated in under two minutes (mostly Kubernetes pod scheduling time) with the same process and PID, no re-launch required.
The approach combines three components: MaxText for the sharded training loop, Pathways for single-controller orchestration, and Orbax for coordinated checkpointing where each TPU host writes its own shard to Cloud Storage in parallel.
THE CLUSTER