LLMs仍然无法规划;LRMs能吗?对OpenAI的o1在PlanBench上的初步评估
LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
AI总结:
本文评估了OpenAI的o1模型在PlanBench基准上的规划能力,发现其性能显著超越现有LLMs但仍未饱和,并指出部署前需考虑准确性、效率和保证问题。
摘要:
规划一系列行动以实现预期状态的能力一直被视为智能体的核心能力,并且自AI研究起步以来就是其不可或缺的组成部分。随着大型语言模型(LLMs)的出现,关于它们是否具备此类规划能力的问题引起了广泛关注。PlanBench是我们在GPT3发布后不久于2022年开发的一个可扩展基准测试,它一直是评估LLMs规划能力的重要工具。尽管自GPT3以来涌现了大量新的私有和开源LLMs,但该基准测试上的进展却出奇地缓慢。OpenAI声称其最近的o1(Strawberry)模型经过专门构建和训练,能够摆脱自回归LLMs的常规限制——使其成为一种新型模型:大型推理模型(LRM)。以这一发展为催化剂,本文全面考察了当前LLMs和新型LRMs在PlanBench上的表现。我们将看到,尽管o1的性能在该基准测试上实现了量子级飞跃,超越了竞争对手,但仍远未达到饱和。这种提升也凸显了关于准确性、效率和保证的问题,在部署此类系统之前必须予以考虑。
英文摘要:
The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities. PlanBench, an extensible benchmark we developed in 2022, soon after the release of GPT3, has remained an important tool for evaluating the planning abilities of LLMs. Despite the slew of new private and open source LLMs since GPT3, progress on this benchmark has been surprisingly slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs--making it a new kind of model: a Large Reasoning Model (LRM). Using this development as a catalyst, this paper takes a comprehensive look at how well current LLMs and new LRMs do on PlanBench. As we shall see, while o1's performance is a quantum improvement on the benchmark, outpacing the competition, it is still far from saturating it. This improvement also brings to the fore questions about accuracy, efficiency, and guarantees which must be considered before deploying such systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。