协作健身房：一种促进和评估人机协作的框架
Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration
AI总结：
Co-Gym框架通过模拟和真实环境支持人机协作研究，展示协作代理在任务性能上的优势，同时揭示当前语言模型的局限性。
摘要：
尽管大语言模型的进步推动了AI代理自动化任务的发展，但许多应用场景本质上需要代理与人类协作，因为人类具有潜在的偏好、领域专业知识或对控制的需求。为了促进人机协作研究，我们引入了协作健身房（Co-Gym），一个开放框架，用于开发和评估能够与人类进行双向交流的协作代理，同时与任务环境互动。我们描述了该框架如何通过灵活的非轮流交互范式实现新的任务环境和人类与代理之间的协调，并通过评估套件评估协作的结果和过程。我们的框架提供了具有可靠用户模拟器的模拟条件和具有交互式网页应用的真实世界条件。在三个代表性任务——制定旅行计划、撰写相关工作部分和分析表格数据上的初始基准实验表明了人机协作的优势：表现最好的协作代理在任务性能上始终优于完全自主的代理，当由真实用户评估时，旅行规划的胜率达到86%，表格分析的胜率达到74%，相关工作的胜率达到66%。尽管有这些改进，我们的评估揭示了当前语言模型和代理的持续限制，观察到在真实条件下，通信和情境意识失败分别出现在65%和40%的案例中。以许可的MIT许可证发布，Co-Gym支持添加新的任务环境，并可用于开发协作代理应用，同时其评估套件可用于评估和改进协作代理。
英文摘要：
While the advancement of large language models has spurred the development of AI agents to automate tasks, numerous use cases inherently require agents to collaborate with humans due to humans' latent preferences, domain expertise, or the need for control. To facilitate the study of human-agent collaboration, we introduce Collaborative Gym (Co-Gym), an open framework for developing and evaluating collaborative agents that engage in bidirectional communication with humans while interacting with task environments. We describe how the framework enables the implementation of new task environments and coordination between humans and agents through a flexible, non-turn-taking interaction paradigm, along with an evaluation suite that assesses both collaboration outcomes and processes. Our framework provides both a simulated condition with a reliable user simulator and a real-world condition with an interactive web application. Initial benchmark experiments across three representative tasks -- creating travel plans, writing related work sections, and analyzing tabular data -- demonstrate the benefits of human-agent collaboration: The best-performing collaborative agents consistently outperform their fully autonomous counterparts in task performance, achieving win rates of 86% in Travel Planning, 74% in Tabular Analysis, and 66% in Related Work when evaluated by real users. Despite these improvements, our evaluation reveals persistent limitations in current language models and agents, with communication and situational awareness failures observed in 65% and 40% of cases in the real condition, respectively. Released under the permissive MIT license, Co-Gym supports the addition of new task environments and can be used to develop collaborative agent applications, while its evaluation suite enables assessment and improvement of collaborative agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
