arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.13285cs.LGcs.AI
收藏

Brittlebench: 通过提示敏感性量化大语言模型的鲁棒性

Brittlebench: Quantifying LLM robustness via prompt sensitivity

FAIR at Meta(Meta FAIR);EPFL(瑞士联邦理工学院洛桑);Northeastern University(东北大学)

Angelika Romanou, Mark Ibrahim, Candace Ross, Kerem Oktar, Chantal Shaib, Samuel J. Bell, Anaelia Ovalle, Antoine Bosselut, Jesse Dodge, Koustuv Sinha, Adina Williams

AI总结:

本文提出Brittlebench框架,通过提示变异量化大语言模型的鲁棒性,揭示模型在真实输入下的性能波动,强调对更鲁棒模型和评估的需求。

摘要:

现有评估方法主要依赖干净静态基准,难以捕捉真实用户输入中的噪声和变异性,从而高估真实模型性能。本文引入理论框架量化模型对提示变异的敏感性,设计Brittlebench评估流程,对主流基准施加语义保持扰动,发现模型性能可能下降12%。然而,这些扰动对模型影响不均,单个扰动会改变63%模型的相对排名,影响模型性能比较结论。分解最先进开源和商用模型的总方差,发现语义保持输入扰动可解释模型性能波动的一半。Brittlebench凸显了对更鲁棒评估和模型的需求,使系统理解模型的脆弱性。

英文摘要:

Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑