Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour

Details

Standard search benchmarks have a critical flaw when evaluating AI search agents: agents equipped with fetch tools can simply download the public answer key mid-evaluation, and models with strong parametric memory can answer from training data rather than retrieved content. NEEDLE sidesteps this entirely by never freezing its query set. News queries regenerate hourly from 124 curated RSS feeds and Google Trends. Finance, scholar, legal, and rare-entity queries refresh daily from SEC XBRL filings, arXiv, Europe PMC, CourtListener opinions, and public agent trajectory logs. The result is a benchmark that's always grading current retrieval performance, not historical contamination. The benchmark covers five verticals—News, Everyday, Expert, Deep-tail, and Legal—each modeled after a different agent intent. Scoring matches the task type: LLM-judged nDCG@5 for open-ended news and rare-entity queries, answer-recall@5 for finance facts, and identifier match for known scholarly papers and court opinions. NEEDLE ships as a Python CLI installable via uv sync and needs an OpenRouter key for LLM judging plus one API key per search engine under test. All 15 APIs receive identical query text, sequential (not concurrent) calls, and results clipped to 2,000 characters—making latency and quality scores directly comparable across engines.