Arena-Lite:通过基于锦标赛的直接比较实现高效可靠的大语言模型评估
Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons
AI总结:
针对现有大语言模型评估依赖基线、可靠性不足的问题,提出融合锦标赛结构与两两直接比较的Arena-Lite框架,经实验验证其以更少比较次数实现更高评估可靠性,还发布了演示与代码助力模型选型。
摘要:
随着大语言模型(LLM)在各领域的应用拓展,LLM评判器已成为系统评估的核心工具。当前基准测试通常将系统输出与基线进行比较。这种依赖基线的方法虽便捷,但可靠性低于系统间的直接比较。我们提出Arena-Lite,在两两比较的基础上整合了锦标赛结构。锦标赛结构与直接比较的应用无需基线输出,减少了所需比较次数,且能提升系统排名的可靠性。我们开展了两项实验:(1)受控随机建模实验;(2)采用真实LLM评判器的实证验证实验。这些实验共同证明,即便使用更小的数据集或能力更弱的评判器,Arena-Lite也能始终以更少的比较次数实现更高的可靠性。我们发布了易用的网页演示版与代码,以推动Arena-Lite的应用,为研究界和产业界的模型选型工作提供便利。Arena-Lite的演示与代码可访问https://huggingface.co/spaces/NCSOFT/ArenaLite获取。
英文摘要:
As Large Language Models (LLMs) expand across domains, LLM judges have become essential for systems evaluation. Current benchmarks typically compare system outputs against baselines. This baseline-mediated approach, though convenient, yields lower reliability than direct comparison between systems. We propose Arena-Lite which integrates tournament structure on top of head-to-head comparison. The application of a tournament structure and direct comparison eliminates the need for baseline outputs, reduces the number of required comparisons, and allows higher reliability in system rankings. We conducted two experiments: (1) controlled stochastic modeling and (2) empirical validation with a real LLM judge. Those experiments collectively demonstrate that Arena-Lite consistently achieves higher reliability with fewer comparisons, even with smaller datasets or weaker judges. We release an easy-to-use web demonstration and code to foster adoption of Arena-Lite, streamlining model selection across research and industry communities. Arena-Lite demo and code are available on \href{https://huggingface.co/spaces/NCSOFT/ArenaLite}{https://huggingface.co/spaces/NCSOFT/ArenaLite}
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。