AI Signal 284
Large-scale reproduction of 2,200 ICML papers finds 23% had falsified or contested claims
↗
A community hackathon used coding agents to reproduce 2,226 ICML 2026 papers, finding that 23% of examined papers had at least one claim falsified or contested, while 51% had at least one claim independently verified.
This demonstrates that coding agents can now attempt scientific reproduction at scale, auditing papers in an afternoon that would cost a human reviewer a weekend. With ICML submissions roughly doubling year-over-year partly due to AI agents, the same technology driving the flood can help verify it.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Over 1,200 participants used coding agents to reproduce 2,226 ICML 2026 papers, producing 6,816 logbooks with 35,908 claims judged.
23% of examined papers had at least one claim falsified or contested, including 49 papers where all claims were falsified.
51% of papers had at least one claim verified, with 266 papers fully reproduced and 632 more partially reproduced with nothing falsified.
THE CLUSTER