arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.07481cs.SE
收藏

大语言模型评估报告的毒圣杯

The Poisoned Chalice of LLM Evaluation Report

Delft University of Technology(代尔夫特理工大学);University of Alberta(阿尔伯塔大学)

Jonathan Katzy, Ali Al-Kaswan, Razvan Mihai Popescu, Zhou Yang

AI总结:

研究大语言模型评估中因数据污染致结果不可靠的问题,通过举办竞赛将污染检测设为白盒成员推理任务,提供相关资源,报告竞赛设置与结果,推动软件工程中可信大语言模型评估社区发展。

摘要:

大语言模型越来越多地用于评估和支持软件工程任务,但其评估的有效性常因基准实例在预训练期间是否被见过的不确定性而受到损害,这可能导致数据污染,夸大性能并得出关于模型能力的误导性结论。许多现代模型的训练语料库仅部分公开,直接净化不可行。为此,我们组织了首届大语言模型评估毒圣杯竞赛,将污染检测框架为对源代码的白盒成员推理任务,为参与者提供精心策划的数据集、目标模型、基线攻击以及对保留模型和数据集的最终评估。本文报告了竞赛的设置和结果,该竞赛旨在推动软件工程中可信大语言模型评估的社区发展。

英文摘要:

Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models' prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting. This paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑