$\tau$-Rec-Books: A Disclosure-Controlled Benchmark for Requirement Elicitation in Agentic Recommender Systems
| Source: arXiv AI
Tags: tau-Rec-Books, conversational recommendation, requirement elicitation, Wikidata, agent benchmark, ECIR 2027
The tau-Rec-Books benchmark shows AI recommender agents drop from 0.975 success when users state all needs upfront to 0.283 when some needs stay hidden, mostly because agents fail to ask or probe.
Details
tau-Rec-Books tests how well agents elicit user requirements before recommending a book. Each request stays fixed while the timing of disclosure varies: all upfront, one upfront and the rest on request, or some never stated. It has 450 tasks over 20,000 Wikidata books, a rule-based simulated user, and scoring against catalog metadata without an LLM judge. Across four agents and two search interfaces, success falls from 0.975 to 0.756 (mixed) and 0.283 (hidden). Mixed tasks fail because agents don't ask; told to ask, all reach at least 0.931. On hidden tasks agents often recommend books they never proposed, and stay below a baseline that uses rejections to rule out requirements.