WorkArena++：迈向基于组合规划与推理的常识性知识工作任务
WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
AI总结：
本文提出WorkArena++基准，包含682个真实知识工作任务，用于评估智能体的规划、推理与检索能力，并通过实证研究揭示现有LLM和VLM在企业场景中的不足，同时提供轨迹生成机制以支持模型微调。
摘要：
大型语言模型（LLMs）模仿人类智能的能力引发了基于LLM的自主智能体的激增。尽管近期LLMs在给定用户指令时似乎能够进行规划和推理，但它们在自主任务求解中应用这些能力的有效性仍未得到充分探索。在企业环境中尤其如此，自动化智能体有望产生巨大影响。为填补这一空白，我们提出了WorkArena++，这是一个新颖的基准测试，包含682个任务，对应知识工作者日常执行的真实工作流程。WorkArena++旨在评估网络智能体的规划、问题求解、逻辑/算术推理、检索和上下文理解能力。我们针对最先进的LLMs和视觉-语言模型（VLMs）以及人类工作者进行的实证研究揭示了此类模型在工作场所充当有用助手所面临的若干挑战。除了基准测试之外，我们还提供了一种机制，可轻松生成数千条真实观测/动作轨迹，用于微调现有模型。总体而言，我们期望这项工作能成为帮助社区朝着有能力自主智能体方向前进的有用资源。该基准测试可在https://github.com/ServiceNow/WorkArena找到。
英文摘要：
The ability of large language models (LLMs) to mimic human-like intelligence has led to a surge in LLM-based autonomous agents. Though recent LLMs seem capable of planning and reasoning given user instructions, their effectiveness in applying these capabilities for autonomous task solving remains underexplored. This is especially true in enterprise settings, where automated agents hold the promise of a high impact. To fill this gap, we propose WorkArena++, a novel benchmark consisting of 682 tasks corresponding to realistic workflows routinely performed by knowledge workers. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents. Our empirical studies across state-of-the-art LLMs and vision-language models (VLMs), as well as human workers, reveal several challenges for such models to serve as useful assistants in the workplace. In addition to the benchmark, we provide a mechanism to effortlessly generate thousands of ground-truth observation/action traces, which can be used for fine-tuning existing models. Overall, we expect this work to serve as a useful resource to help the community progress toward capable autonomous agents. The benchmark can be found at https://github.com/ServiceNow/WorkArena.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
