以陪审团取代法官：用多样化模型小组评估大语言模型的生成结果
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
AI总结：
本文提出用多个较小模型组成评估小组（PoLL）替代单一大型模型法官，在六个数据集上验证其评估效果更优、模型内偏差更小且成本降低七倍以上。
摘要：
随着大语言模型（LLMs）日益先进，我们准确评估其质量的能力已跟不上其发展速度。不仅寻找能够充分探测特定模型属性的数据十分困难，仅评估模型自由形式生成的正确性本身也是一项挑战。为解决这一问题，许多评估工作如今依赖将大语言模型本身用作法官，对其他大语言模型输出的质量进行评分。评估最常使用像GPT4这样的单一大型模型。尽管这种方法日益流行，但它成本高昂，已被证明会引入模型内偏差，并且在本工作中，我们发现非常大的模型往往并非必要。我们提出改用一个大语言模型评估者小组（PoLL）来评估模型。在三种不同的法官设置下，并跨越六个不同的数据集，我们发现，使用由更多较小模型组成的PoLL优于单一大型法官，由于其由不相交的模型家族构成而表现出更少的模型内偏差，同时成本降低了七倍以上。
英文摘要：
As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
