Evalci:用于语言模型评估统计严格比较的Python库
evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations
AI总结:
研究语言模型评估中仅依单一准确率数字判断优劣的问题,用evalci库通过统计方法处理结果表,给出严谨结论,如置信区间等,还通过案例分析发现部分排行榜差距无统计学意义。
摘要:
语言模型评估的主流做法是为每个模型报告一个单一的准确率数字,并宣称数字越高越好,而不测试差距是否可能是抽样噪声。在几千个项目的基准测试中,以及在温度采样下,这种做法经常高估对头条声明的信心。我们提出了evalci,一个纯Python库,它可以将每个项目的结果表转化为一个可发表的声明。作为一个案例研究,我们重新分析了九个语言模型的MMLU准确率的公开比较,发现8个相邻排行榜排名差距中的3个在对排名所隐含的36对比较进行校正后没有统计学意义。
英文摘要:
The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B, $Δ=3.1$ pts, 95% CI [1.2, 5.0], paired permutation $p=0.002$, $n=1{,}319$" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。