arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.00696cs.CLcs.AI
收藏

Polyrating:面向大语言模型评估的高性价比且偏差感知的评分系统

Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

ETH Zurich(苏黎世联邦理工学院)

Jasper Dekoninck, Maximilian Baader, Martin Vechev

AI总结:

针对现有LLM评分系统存在偏差未考虑、成本高、跨任务不可比的问题,提出基于最大后验估计的Polyrating系统,可检测量化偏差、降低41%-77%评估成本并支持跨任务比较,实现更细致的低成本模型性能分析。

摘要:

基于评分的人工评估已成为准确评估大型语言模型(LLMs)出色性能的重要工具。然而,当前评分系统存在若干关键局限性:首先,它们未能考虑对评估结果有显著影响的偏差;其次,需要庞大且昂贵的偏好数据集才能获得准确评分;第三,无法促进不同任务间模型评分的有意义比较。为解决这些问题,我们引入Polyrating——一种基于最大后验估计的富有表现力且灵活的评分系统,能以更低成本对模型性能进行更细致、全面的分析。Polyrating可检测并量化影响人类偏好的偏差,确保更公平的模型比较;通过利用现有基准分数,新模型评估成本可降低多达41%,新任务评估成本可降低多达77%;还能实现不同任务间评分的直接比较,全面了解LLMs在不同应用中的优势、劣势及相对性能。

英文摘要:

Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first, they fail to account for biases that significantly influence evaluation results, second, they require large and expensive preference datasets to obtain accurate ratings, and third, they do not facilitate meaningful comparisons of model ratings across different tasks. To address these issues, we introduce Polyrating, an expressive and flexible rating system based on maximum a posteriori estimation that enables a more nuanced and thorough analysis of model performance at lower costs. Polyrating can detect and quantify biases affecting human preferences, ensuring fairer model comparisons. Further, Polyrating can reduce the cost of human evaluations by up to $41\%$ for new models and up to $77\%$ for new tasks by leveraging existing benchmark scores. Lastly, Polyrating enables direct comparisons of ratings across different tasks, providing a comprehensive understanding of an LLMs' strengths, weaknesses, and relative performance across different applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑