语言模型委员会：针对高主观性任务的基础模型民主式基准测试
Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks
AI总结：
针对单一LLM裁判评估高主观性任务存在偏差的问题，提出由多LLM组成的民主式评估框架Language Model Council，经情绪智力案例验证其排名更鲁棒且更贴合人类判断，并探讨了成本优化方案。
摘要：
随着大语言模型（LLM）持续演进，对其的评估仍是一项长期存在的挑战。近期许多评估采用LLM作为裁判来为其他LLM的输出打分，且通常依赖GPT-4o这类单一大模型。然而，使用单一LLM裁判易受模型内部偏差影响，且许多任务——比如与情绪智力、创意写作和说服力相关的任务——主观性过强，单一模型无法做出公正评判。我们提出语言模型委员会（Language Model Council, LMC），由一组LLM协作设计测试题目、作答并互评彼此的回答，以民主方式生成排名。与以往通过小型模型面板来降低成本或减少偏差的方法不同，本研究探究了完全包容性LLM评估系统的优势与细微差异。在一项关于情绪智力的详细案例研究中，我们组建了包含20个近期LLM的委员会，针对人际冲突场景的开放式回答对各模型进行排名。结果表明，LMC生成的排名区分度更高、鲁棒性更强；用户研究显示，与任何单个LLM裁判相比，其排名与人类评估的一致性更高。不过，使用所有LLM担任裁判成本较高，因此我们通过蒙特卡洛模拟和人工筛选的子委员会研究了假设的委员会构成，并探讨了增量式LLM裁判的价值。
英文摘要：
As Large Language Models (LLMs) continue to evolve, evaluating them remains a persistent challenge. Many recent evaluations use LLMs as judges to score outputs from other LLMs, often relying on a single large model like GPT-4o. However, using a single LLM judge is prone to intra-model bias, and many tasks - such as those related to emotional intelligence, creative writing, and persuasiveness - may be too subjective for a single model to judge fairly. We introduce the Language Model Council (LMC), where a group of LLMs collaborate to create tests, respond to them, and evaluate each other's responses to produce a ranking in a democratic fashion. Unlike previous approaches that focus on reducing cost or bias by using a panel of smaller models, our work examines the benefits and nuances of a fully inclusive LLM evaluation system. In a detailed case study on emotional intelligence, we deploy a council of 20 recent LLMs to rank each other on open-ended responses to interpersonal conflicts. Our results show that the LMC produces rankings that are more separable and more robust, and through a user study, we show that they are more consistent with human evaluations than any individual LLM judge. Using all LLMs for judging can be costly, however, so we use Monte Carlo simulations and hand-curated sub-councils to study hypothetical council compositions and discuss the value of the incremental LLM judge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
