arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.18326cs.CLcs.AI
收藏

PaCoST:用于大型语言模型基准污染检测的配对置信度显著性检验

PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Foreign Languages, Peking University(北京大学外国语学院)

Huixuan Zhang, Yun Lin, Xiaojun Wan

AI总结:

针对LLM基准污染导致评测失真的问题,提出PaCoST配对置信度显著性检验方法,通过同分布对应项与置信度统计分析检测污染,并在开源模型和基准上验证。

摘要:

众所周知,大型语言模型(LLM)是在海量数据上训练的,这些数据可能无意或有意地包含来自常用基准的数据。这种纳入会导致模型排行榜上出现作弊般的高分,但在实际应用中却表现得令人失望。为了解决这一基准污染问题,我们首先提出了一套实用的污染检测方法应当遵循的要求。遵循这些提出的要求,我们引入了 PaCoST,这是一种配对置信度显著性检验,可有效检测 LLM 中的基准污染。我们的方法为每条数据构造一个具有相同分布的对应项,并对相应置信度进行统计分析,以检验模型在原始基准下是否显著更自信。我们验证了 PaCoST 的有效性,并将其应用于流行的开源模型和基准。我们发现,几乎所有测试过的模型和基准都或多或少存在疑似污染。我们最后呼吁采用新的 LLM 评估方法。

英文摘要:

Large language models (LLMs) are known to be trained on vast amounts of data, which may unintentionally or intentionally include data from commonly used benchmarks. This inclusion can lead to cheatingly high scores on model leaderboards, yet result in disappointing performance in real-world applications. To address this benchmark contamination problem, we first propose a set of requirements that practical contamination detection methods should follow. Following these proposed requirements, we introduce PaCoST, a Paired Confidence Significance Testing to effectively detect benchmark contamination in LLMs. Our method constructs a counterpart for each piece of data with the same distribution, and performs statistical analysis of the corresponding confidence to test whether the model is significantly more confident under the original benchmark. We validate the effectiveness of PaCoST and apply it on popular open-source models and benchmarks. We find that almost all models and benchmarks we tested are suspected contaminated more or less. We finally call for new LLM evaluation methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑