arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01622cs.CL
收藏

TravelPlanner:面向语言智能体真实世界规划任务的基准

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

Fudan University(复旦大学);The Ohio State University(俄亥俄州立大学);The Pennsylvania State University(宾夕法尼亚州立大学);Meta AI

Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su

AI总结:

本文提出 TravelPlanner 基准,通过旅行规划任务评估语言智能体在真实世界复杂规划中的能力,发现当前模型表现极差,GPT-4 成功率仅 0.6%,为未来研究提供了挑战性测试平台。

摘要:

自人工智能概念诞生以来,规划一直是其核心追求之一,但早期的 AI 智能体大多聚焦于受限环境,因为实现人类水平规划所需的许多认知基础仍然缺失。近年来,由大语言模型(LLMs)驱动的语言智能体展现出了工具使用和推理等有趣的能力。这些语言智能体是否能够在更复杂、超出以往 AI 智能体能力范围的环境中完成规划任务?为了推进这一研究,我们提出了 TravelPlanner,这是一个专注于旅行规划这一常见真实世界规划场景的新基准。它提供了一个丰富的沙盒环境、用于访问近四百万条数据记录的多种工具,以及 1,225 个精心策划的规划意图和参考规划。综合评估表明,当前的语言智能体尚不具备处理此类复杂规划任务的能力——即便是 GPT-4 也仅取得了 0.6% 的成功率。语言智能体在保持任务专注、使用正确工具收集信息以及跟踪多重约束条件方面都存在困难。然而,我们注意到,语言智能体能够尝试处理如此复杂的问题这一可能性本身,就已经是非平凡的进展。TravelPlanner 为未来的语言智能体提供了一个具有挑战性且富有意义的测试平台。

英文摘要:

Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑