通过可评分游戏评估大语言模型在谈判中的能力基准测试
[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games
AI总结:
本文通过可评分游戏评估大语言模型在谈判中的能力,发现基准复杂但模型比较存在歧义,强调上下文在评估中的重要性。
摘要:
大语言模型(LLMs)在多智能体谈判任务中展现出显著潜力,但该领域评估仍面临挑战,因缺乏稳健且可推广的基准。Abdelnabi等人(2024)引入了一个基于可评分游戏的谈判基准,旨在为LLMs开发一个高度复杂且真实的评估框架。我们的工作研究了其基准声明的可复现性,并提供了对其可用性和可推广性的深入理解。我们将在其他模型上复现原始实验,并引入额外指标以验证谈判质量和评估的公平性。我们的发现表明,尽管该基准确实复杂,但模型比较存在歧义,引发了对其客观性的质疑。此外,我们识别了实验设置中的局限性,特别是信息泄漏检测和消融研究的彻底性。通过检查和分析更广泛模型在基准扩展版上的行为,我们揭示了见解,为潜在用户提供了额外的背景。我们的结果突显了在模型比较评估中上下文的重要性。
英文摘要:
Large Language Models (LLMs) demonstrate significant potential in multi-agent negotiation tasks, yet evaluation in this domain remains challenging due to a lack of robust and generalizable benchmarks. Abdelnabi et al. (2024) introduce a negotiation benchmark based on Scoreable Games, with the aim of developing a highly complex and realistic evaluation framework for LLMs. Our work investigates the reproducibility of claims in their benchmark, and provides a deeper understanding of its usability and generalizability. We replicate the original experiments on additional models, and introduce additional metrics to verify negotiation quality and evenness of evaluation. Our findings reveal that while the benchmark is indeed complex, model comparison is ambiguous, raising questions about its objectivity. Furthermore, we identify limitations in the experimental setup, particularly in information leakage detection and thoroughness of the ablation study. By examining and analyzing the behavior of a wider range of models on an extended version of the benchmark, we reveal insights that provide additional context to potential users. Our results highlight the importance of context in model-comparative evaluations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。