HotelQuEST: 在代理搜索中平衡质量和效率
HotelQuEST: Balancing Quality and Efficiency in Agentic Search
AI总结:
HotelQuEST 是一个用于评估代理搜索系统质量和效率的基准测试,通过包含 214 个酒店查询,揭示了基于 LLM 的代理在准确性上的优势,但成本较高,需进一步优化。
摘要:
代理搜索作为一种由大型语言模型(LLMs)驱动的自适应检索系统的新 paradigm,已经展现出巨大的潜力。然而,现有的基准测试主要关注质量,忽略了效率因素,这些因素对实际应用至关重要。此外,现实中的用户查询往往包含不明确的偏好,这一挑战在当前的代理搜索评估中仍鲜有研究。因此,尽管代理搜索系统表现出色,但许多系统仍然不实用。在本工作中,我们介绍了 HotelQuEST,一个包含 214 个酒店搜索查询的基准测试,这些查询从简单的事实请求到复杂的查询不等,使评估能够覆盖查询难度的整个范围。我们进一步通过收集澄清来解决评估不明确用户偏好的挑战,使标注者的隐含偏好显性化以供评估。我们发现,基于 LLM 的代理在准确性上优于传统检索器,但由于冗余的工具调用和不匹配查询复杂性与模型能力的路由,成本显著更高。我们的分析揭示了当前代理搜索系统的低效性,并展示了对成本敏感优化的显著潜力。
英文摘要:
Agentic search has emerged as a promising paradigm for adaptive retrieval systems powered by large language models (LLMs). However, existing benchmarks primarily focus on quality, overlooking efficiency factors that are critical for real-world deployment. Moreover, real-world user queries often contain underspecified preferences, a challenge that remains largely underexplored in current agentic search evaluation. As a result, many agentic search systems remain impractical despite their impressive performance. In this work, we introduce HotelQuEST, a benchmark comprising 214 hotel search queries that range from simple factual requests to complex queries, enabling evaluation across the full spectrum of query difficulty. We further address the challenge of evaluating underspecified user preferences by collecting clarifications that make annotators' implicit preferences explicit for evaluation. We find that LLM-based agents achieve higher accuracy than traditional retrievers, but at substantially higher costs due to redundant tool calls and suboptimal routing that fails to match query complexity to model capability. Our analysis exposes inefficiencies in current agentic search systems and demonstrates substantial potential for cost-aware optimization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。