PERFORMANCE Signal 129
The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents
↗
ClickHouse open-sourced a benchmarking framework to evaluate 28 models on real analytics questions from its production data warehouse, measuring correctness, speed, and cost.
Agentic analytics systems require models to dynamically discover schemas and execute multi-step queries, unlike static text-to-SQL benchmarks. This framework lets engineers test models against their own data warehouses to identify the best trade-off between accuracy and cost. The results show significant cost disparities between models with only marginal accuracy differences.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
The benchmark uses 201 real analytics questions from ClickHouse’s production traffic, testing models on correctness, speed, and cost.
Frontier-level models dominate the top 10 for correctness, but open-weight models are competitive in cost and performance.
Running the full benchmark costs vary widely, from $1 to $52, with only an 11 percentage-point difference in correctness between extremes.
THE CLUSTER