clembench-2024：一个面向作为多动作智能体的LLM的具有挑战性、动态、互补、多语言的基准及底层灵活框架
clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents
AI总结：
本文扩展一个用于LLM自我博弈对话游戏的框架，验证其作为多维度评估工具的有效性，为模型选择和闭环开发环境奠定基础。
摘要：
近期研究已表明，大型语言模型（LLMs）可被提示进行“自我博弈”的对话游戏，以探究某些能力（通用指令遵循、策略性目标导向、语言理解能力），由此产生的交互式游戏过程可被自动评分。在本文中，我们采用其中一个用于设置此类游戏环境的框架，并从多个维度进一步检验其作为评估工具的有效性：我们证明它能轻松跟上新进展同时避免数据污染；我们证明其中实现的测试尚未饱和（人类表现显著优于即使是最佳模型的表现）；我们证明它适用于研究其他问题，如提示语言对性能的影响。我们相信，该方法为针对构建应用交互系统进行模型选择决策，乃至最终建立系统与模拟评估器的闭环开发环境，奠定了良好基础。
英文摘要：
It has been established in recent work that Large Language Models (LLMs) can be prompted to "self-play" conversational games that probe certain capabilities (general instruction following, strategic goal orientation, language understanding abilities), where the resulting interactive game play can be automatically scored. In this paper, we take one of the proposed frameworks for setting up such game-play environments, and further test its usefulness as an evaluation instrument, along a number of dimensions: We show that it can easily keep up with new developments while avoiding data contamination, we show that the tests implemented within it are not yet saturated (human performance is substantially higher than that of even the best models), and we show that it lends itself to investigating additional questions, such as the impact of the prompting language on performance. We believe that the approach forms a good basis for making decisions on model choice for building applied interactive systems, and perhaps ultimately setting up a closed-loop development environment of system and simulated evaluator.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
