从温和到动态对抗的评估：一个大型语言模型的鱿鱼游戏
Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models
AI总结：
本文提出鱿鱼游戏，一个动态对抗性评估环境，用于评估大型语言模型的多方面能力，并揭示静态基准中可能存在的评估范式污染问题。
摘要：
当代大型语言模型（LLMs）基准测试中潜在的数据污染问题提出了建立可信评估框架的根本挑战。同时，它们主要假设温和、资源丰富的环境，留下了LLM在压力下的行为未被探索。在本文中，我们引入了鱿鱼游戏，一个动态且对抗性的评估环境，通过互动游戏对抗其他LLM对手来评估LLM。鱿鱼游戏包含六个淘汰式关卡，专注于多方面的技能，包括指令遵循、代码、推理、规划和安全对齐。我们在鱿鱼游戏中评估了超过50个LLM，展示了对动态对抗场景下通用LLM的最全面的行为评估研究。我们观察到同一模型系列表现中存在明显的代际相变，并发现某些模型会采用推测捷径来赢得游戏，表明静态基准中可能存在更高层次的评估范式污染。我们还比较了主要的LLM基准和鱿鱼游戏，指出动态评估可以作为静态评估的补充部分。项目页面：https://github.com/zijianchen98/LLM_Squid_Game。
英文摘要：
The potential data contamination issue in contemporary large language models (LLMs) benchmarks presents a fundamental challenge to establishing trustworthy evaluation frameworks. Meanwhile, they predominantly assume benign, resource-rich settings, leaving the behavior of LLMs under pressure unexplored. In this paper, we introduce \textsc{Squid Game}, a dynamic and adversarial evaluation environment with resource-constrained and asymmetric information settings elaborated to evaluate LLMs through interactive gameplay against other LLM opponents. Squid Game consists of six elimination-style levels, focusing on multi-faceted abilities, including instruction-following, code, reasoning, planning, and safety alignment. We evaluate over 50 LLMs on Squid Game, presenting the largest behavioral evaluation study of general LLMs on dynamic adversarial scenarios. We observe a clear generational phase transition in performance in the same model lineage and find evidence that some models resort to speculative shortcuts to win the game, indicating the possibility of higher-level evaluation paradigm contamination in static benchmarks. We also compare prominent LLM benchmarks and \textsc{Squid Game}, highlighting that dynamic evaluation can serve as a complementary part for static evaluations. Project page: https://github.com/zijianchen98/LLM_Squid_Game.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
