arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12424cs.CLcs.AI
收藏

RankLLM: 通过量化问题难度对大型语言模型进行加权排名

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

Lehigh University(莱维大学);University of Notre Dame(诺特大学);Zhejiang Wanli University(浙江万里大学);Squirrel Ai Learning;City University of Hong Kong(香港城市大学);Duke University(杜克大学)

Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun

AI总结:

提出RankLLM框架,通过量化问题难度和模型能力实现细粒度评估,在35550个问题上对30个模型进行评测,与人类判断一致性达90%。

摘要:

基准测试建立了标准化的评估框架,以系统评估大型语言模型(LLM)的性能,促进客观比较并推动该领域的进步。然而,现有基准测试未能区分问题难度,限制了其有效区分模型能力的能力。为解决这一局限,我们提出了RankLLM,一种旨在量化问题难度和模型能力的新框架。RankLLM引入难度作为区分的主要标准,实现了对LLM能力的更细粒度评估。RankLLM的核心机制促进了模型与问题之间的双向分数传播。RankLLM的核心直觉是:当模型正确回答一个问题时,它获得一个能力分数;而当一个问题难倒模型时,其难度分数增加。利用该框架,我们在多个领域的35550个问题上评估了30个模型。RankLLM与人类判断的一致性达到90%,并且始终优于IRT等强基线。它还表现出强大的稳定性、快速收敛和高计算效率,使其成为大规模、难度感知的LLM评估的实用解决方案。

英文摘要:

Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑