自人工智能概念诞生以来,规划一直是其核心追求之一,但早期的 AI 智能体大多聚焦于受限环境,因为实现人类水平规划所需的许多认知基础仍然缺失。近年来,由大语言模型(LLMs)驱动的语言智能体展现出了工具使用和推理等有趣的能力。这些语言智能体是否能够在更复杂、超出以往 AI 智能体能力范围的环境中完成规划任务?为了推进这一研究,我们提出了 TravelPlanner,这是一个专注于旅行规划这一常见真实世界规划场景的新基准。它提供了一个丰富的沙盒环境、用于访问近四百万条数据记录的多种工具,以及 1,225 个精心策划的规划意图和参考规划。综合评估表明,当前的语言智能体尚不具备处理此类复杂规划任务的能力——即便是 GPT-4 也仅取得了 0.6% 的成功率。语言智能体在保持任务专注、使用正确工具收集信息以及跟踪多重约束条件方面都存在困难。然而,我们注意到,语言智能体能够尝试处理如此复杂的问题这一可能性本身,就已经是非平凡的进展。TravelPlanner 为未来的语言智能体提供了一个具有挑战性且富有意义的测试平台。
英文摘要:
Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.
IMAGINE: Integrating Multi-Agent System into One Model for Complex Reasoning and Planning
Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Kuaishou Technology(快手科技)
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Tongji University(同济大学);Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);Fudan University(复旦大学);Skywork AI;University of California, Santa Cruz(加州大学圣克鲁兹分校);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)
HiMAP-Travel: Hierarchical Multi-Agent Planning for Long-Horizon Constrained Travel
Department of XXX, University of YYY, Location, Country(YYY大学XXX系);School of ZZZ, Institute of WWW, Location, Country(WWW研究所ZZZ学院);Singapore Management University(新加坡管理大学)
Behavior2Trip: Towards Personalized Travel Planning via User Behavior Trajectory
Meituan Inc.(美团公司);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Beijing Institute of Technology(北京理工大学);University of Edinburgh(爱丁堡大学)
LARC: Towards Human-level Constrained Retrosynthesis Planning through an Agentic Framework
Department of Computer Science and Engineering, 2 Division of Medicinal Chemistry & Pharmacognosy, 3 Department of Biomedical Informatics, 4 Translational Data Analytics Institute(1 计算机科学与工程系,2 药物化学与药理学系,3 生物医学信息学系,4 转化数据分析研究所)
LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics
Automation and Control Institute, Faculty of Electrical Engineering, TU Wien(自动化与控制研究所,电气工程学院,维也纳技术大学);AIT Austrian Institute of Technology GmbH, Center for Vision, Automation and Control(奥地利技术研究所(AIT)有限公司,视觉、自动化与控制中心)