TECH Signal 373
MOSAIC folds cluster throughput into scaling laws, making sparse-MoE picks invert on GPU-hours
↗
A new framework, MOSAIC, optimizes model architecture, token budget, and parallel layout jointly against a cluster's deliverable FLOPs, showing that FLOPs-optimal designs can be the slowest on real hardware.
Scaling laws that ignore GPU-hours mislead pretraining choices: a design that wins on loss per FLOP can be 1.70× slower on a 512-GPU cluster. MOSAIC makes the cluster's throughput part of the optimization, so engineers can pick configurations that actually finish in their training window.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Under a fixed model-FLOPs budget, predicted loss falls monotonically with sparsity across the entire calibrated range, with no interior optimum.
At a world size of 512 GPUs, wall-clock per delivered model FLOP rises monotonically with sparsity, up to 1.70× the densest configuration.
MOSAIC replaces the model-FLOPs budget with deliverable FLOPs, solving a bilevel problem that maximizes MFU over parallel layouts while minimizing predicted loss.
THE CLUSTER