发表机构
Shanghai Jiao Tong University; Tencent（上海交通大学; 腾讯）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究大语言模型能否利用其他模型的未知进行评估，提出自动化抗污染评估框架LivingArena，模型轮流提问，经评审验证，能产生稳定排行榜，可衡量事实严谨性等，与人类偏好相关性弱，提供低成本持续评估方法。
AI 中文摘要
评估前沿大语言模型具有挑战性：静态基准测试存在污染和饱和问题，这使得用户无法区分顶级模型，开发者也对特定的失败模式视而不见，而人类偏好又具有主观性。在本文中，我们提出的问题是：“大语言模型知道其他大语言模型所不知道的吗？我们能否利用这种动态性进行评估？”我们展示了LivingArena，这是一个自动化的、抗污染的评估框架。在这个框架中，模型轮流提出问题，旨在提出对手无法正确回答的项目。提问者被鼓励积极识别和利用对手的知识边界，当回答者失败时获得奖励，否则回答者获得奖励。为确保问题包含可客观验证的答案，由强大模型组成的评审小组对问题进行验证，如果验证失败则惩罚提问者。对十个前沿大语言模型进行评估，LivingArena产生了一个稳定的Elo排行榜。我们的行为分析表明，模型能够识别并利用其对等模型的认知边界：自我博弈和锦标赛日志表明它们能够定位并强化对手的薄弱维度。除了静态知识回忆，对等探测还衡量事实严谨性以及探测对手弱点的高阶能力，与人类偏好的相关性较弱，并提供了一种可扩展、低成本的持续评估方法。
英文摘要
Fixed benchmarks are costly to renew and cannot adapt their questions to model-specific failures. We ask whether LLMs can instead discover one another's weaknesses and turn those observations into an evaluation process. To study this question, we introduce \textbf{LivingArena}, an automated peer-probing framework in which models take turns testing one another. Using the interaction history, each questioner identifies potential weaknesses of its opponent and constructs targeted, verifiable questions to probe them. A 3,600-round tournament of ten models reveals a clear role asymmetry: strong answerers are not always reliable questioners, because they may generate internally inconsistent tests or fail to verify their own reference answers. After a questioner exposes an answerer's failure, it is more likely to pursue the same capability domain, while the answerer's weakness recurs on independently generated questions, including questions written by different models. These findings show that peer probing can reveal persistent model-specific weaknesses while separately evaluating answering and reliable test construction. By automating this process and allowing test difficulty to evolve with model capabilities, LivingArena provides a "living" benchmark for model development, red-teaming, and capability-aware multi-agent coordination. We publicly release our code: https://github.com/galaxyChen/LivingArena
