What Happens When the Model Eats the Stack? Rethinking the Research Agenda for Data Agents to Withstand the Bitter Lesson

elseif has not written about this yet · Metadata describes it this way

General methods that scale with computation will inevitably displace hand-engineered domain knowledge! Sutton’s Bitter Lesson hangs like a Sword of Damocles over all of us. This paper, which just dropped, applies that lesson to data agents, and says that as LLMs improve, they will quickly absorb the agent scaffolding researchers have spent the last few years painstakingly building. It argues that researchers should instead work on building curated contextual information about the data environment (aka. persistent semantic context) to help data agents be efficient across many queries.The Bitter Lesson for Data AgentsThe evaluation section aims to capture the Bitter Lesson in action. The authors compared general coding agents (the Codex harness with no task-specific engineering) against state-of-the-art human-designed data agents on two benchmarks, TAG-Bench and DAB. They use the same models on both sides, so the only variable is the scaffolding.They find that:Although the human-designed Agentar-Scale-SQL won on accuracy and token efficiency with o3, when using GPT-5.6 Sol that flips and the plain coding agent surpasses on both counts. Bam!The number of back-and-forth turns an agent