arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.07445cs.CLcs.AIcs.LG
收藏

忘掉你对LLM评估的认知——LLM就像变色龙

Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleon

Ben Gurion University(本· Gurion 大学);Tel Aviv University(特拉维夫大学);Harvard University(哈佛大学)

Nurit Cohen-Inger, Yehonatan Elisha, Bracha Shapira, Lior Rokach, Seffi Cohen

AI总结:

提出C-BOD框架通过参数变换扭曲基准提示检测LLMs过拟合,发现26个模型中20个在扰动下性能显著下降,挑战社区关注模型泛化能力。

摘要:

大型语言模型(LLMs)通常在公开基准上表现优异,但这些高分可能掩盖了其对数据集特定表面线索的过度依赖,而非真正的语言理解。我们提出了Chameleon Benchmark Overfit Detector(C-BOD),这是一个元评估框架,通过参数变换系统性地扭曲基准提示,并检测LLMs的过拟合情况。通过在保留语义内容和标签的前提下对输入进行改写,C-BOD揭示了模型性能是否由记忆模式驱动。在MMLU基准上使用26个领先LLMs进行评估,我们的方法显示在适度扰动下平均性能下降2.15%,26个模型中有20个表现出统计学显著差异。值得注意的是,基线准确率较高的模型在扰动下表现出更大的性能差异,较大的LLMs对改写更敏感,表明这两种情况都可能过度依赖固定的提示模式。相比之下,Llama家族和基线准确率较低的模型表现出不显著的退化,表明对表面线索的依赖减少。此外,C-BOD与数据集和模型无关的设计使其易于集成到训练流程中,以促进更稳健的语言理解。我们的发现挑战社区超越排行榜分数,优先考虑LLM评估中的韧性和泛化能力。

英文摘要:

Large language models (LLMs) often appear to excel on public benchmarks, but these high scores may mask an overreliance on dataset-specific surface cues rather than true language understanding. We introduce the Chameleon Benchmark Overfit Detector (C-BOD), a meta-evaluation framework that systematically distorts benchmark prompts via a parametric transformation and detects overfitting of LLMs. By rephrasing inputs while preserving their semantic content and labels, C-BOD exposes whether a model's performance is driven by memorized patterns. Evaluated on the MMLU benchmark using 26 leading LLMs, our method reveals an average performance degradation of 2.15% under modest perturbations, with 20 out of 26 models exhibiting statistically significant differences. Notably, models with higher baseline accuracy exhibit larger performance differences under perturbation, and larger LLMs tend to be more sensitive to rephrasings, indicating that both cases may overrely on fixed prompt patterns. In contrast, the Llama family and models with lower baseline accuracy show insignificant degradation, suggesting reduced dependency on superficial cues. Moreover, C-BOD's dataset- and model-agnostic design allows easy integration into training pipelines to promote more robust language understanding. Our findings challenge the community to look beyond leaderboard scores and prioritize resilience and generalization in LLM evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑