你的基准仍然有用吗？代码语言模型的动态基准测试
Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models
AI总结：
研究模型评估中如何在模型训练可能已见过基准的情况下保持其有用性，提出动态基准测试框架，通过语义保留突变变换输入构建新基准，评估发现模型性能变差、排名变化及能抵抗数据污染问题。
摘要：
在本文中，我们解决了模型评估中的一个关键挑战：当模型在训练期间可能已经见过代码基准时，如何保持代码基准的有用性。我们引入了一种新颖的解决方案，即动态基准测试框架，以应对这一挑战。给定一个代码理解或推理基准，我们的框架通过各种语义保留突变动态地变换每个输入，即程序，以构建一个语法上新的但语义上相同的基准。我们在动态基准上评估了10个流行的语言模型。我们的评估揭示了几个有趣或令人惊讶的发现：（1）所有模型的性能都比以前显著变差，（2）一些模型之间的排名发生了巨大变化，（3）动态基准可以抵抗数据污染问题。
英文摘要：
In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework, to address this challenge. Given a code understanding or reasoning benchmark, our framework dynamically transforms each input, i.e., programs, with various semantic-preserving mutations to build a syntactically new while semantically identical benchmark. We evaluated 10 popular language models on our dynamic benchmarks. Our evaluation reveals several interesting or surprising findings: (1) all models perform significantly worse than before, (2) the ranking between some models shifts dramatically, and (3) dynamic benchmarks can resist against the data contamination problem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
