PERFORMANCE Signal 495
Keenable introduces NEEDLE, a live open-source benchmark for agentic web search that resists memorization
↗
Keenable has released NEEDLE, a continuously running, open-source benchmark for search engine quality that draws queries from real agent search logs to avoid the overfitting and data leakage problems plaguing static benchmarks like BrowseComp.
Static search benchmarks can be gamed by models that memorize answers or even fetch the benchmark's own answer key from HuggingFace during evaluation, making existing quality measurements unreliable for agentic workloads. NEEDLE addresses this by refreshing queries continuously, so there is nothing fixed to memorize or leak, giving engineers a more trustworthy signal when comparing search engines for agent traffic.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
NEEDLE queries are drawn from real agent search logs and generated queries reflecting production search intents, running continuously in public with full evaluation code on GitHub.
Existing static benchmarks like BrowseComp are vulnerable to overfitting, memorization, and direct data leakage where agents download answer keys from HuggingFace mid-evaluation.
NEEDLE follows the live-benchmark pattern already established in other domains by LiveBench, LiveCodeBench, and SWE-bench-Live, but is the first such benchmark for web search.
THE CLUSTER