通过谈判评估语言模型的代理能力
Evaluating Language Model Agency through Negotiations
AI总结：
通过谈判游戏评估语言模型的代理能力，发现封闭源模型更擅长完成任务，合作谈判游戏最具挑战性，且强模型可能输给弱对手。
摘要：
我们提出了一种通过谈判游戏评估语言模型（LM）代理能力的方法。该方法更符合现实应用场景，并解决了其他LM基准测试的不足。谈判游戏使我们能够研究多轮和跨模型交互，调节复杂性，并避免偶然的评估数据泄露。我们使用该方法测试了六个广泛使用的公开可用LM，评估其在自play和cross-play设置中的性能和对齐情况。值得注意的发现包括：（i）只有在此处测试的封闭源模型才能完成这些任务；（ii）合作谈判游戏对模型来说最具挑战性；（iii）即使是最强大的模型有时也会输给较弱的对手。
英文摘要：
We introduce an approach to evaluate language model (LM) agency using negotiation games. This approach better reflects real-world use cases and addresses some of the shortcomings of alternative LM benchmarks. Negotiation games enable us to study multi-turn, and cross-model interactions, modulate complexity, and side-step accidental evaluation data leakage. We use our approach to test six widely used and publicly accessible LMs, evaluating performance and alignment in both self-play and cross-play settings. Noteworthy findings include: (i) only closed-source models tested here were able to complete these tasks; (ii) cooperative bargaining games proved to be most challenging to the models; and (iii) even the most powerful models sometimes "lose" to weaker opponents
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
