TECH Signal 243 2 feeds carried it
44% on ARC-AGI-1 in 67 cents
↗
Illustration only Photo by Declan Sun on Unsplash
A small transformer trained from scratch at test time achieves 44% on the ARC-AGI-1 benchmark for 67 cents in compute, matching or exceeding many larger models.
This demonstrates that sample-efficient training on small transformers can compete with far more expensive approaches on metalearning benchmarks. The low cost and open-source availability lower the barrier for researchers to iterate on ARC-AGI problems. The finding that supervised training outperforms unsupervised training despite worse test loss exposes a failure mode in using validation loss as a proxy for benchmark performance.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
A small transformer with modern architecture and test-time training achieves 44% on ARC-AGI-1 for 67 cents, beating many LLMs.
Switching from unsupervised to supervised training improved scores from approximately 40% to 44% despite worse test loss.
3D RoPE embeddings and per-task embeddings are the largest contributors to performance, with ablations dropping scores to 25% without them.
THE CLUSTER