Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models
Geodesic Research;Technical University of Munich(慕尼黑技术大学);Tufts University(塔夫茨大学);SecureBio;Apart Research;Anthropic;Apollo Research
Cameron Tice, Philipp Alexander Kreer, Nathan Helm-Burger, Prithviraj Singh Shahani, Fedor Ryzhenkov, Fabien Roger, Clement Neo, Jacob Haimes, Felix Hofstätter, Teun van der Weij
能力评估在评估和监管前沿AI系统中起着至关重要的作用。这些评估的有效性面临重大挑战:战略性欠佳,或称为“沙袋”现象,其中模型在评估期间故意表现不佳。沙袋行为可能通过显式的开发者干预或通过意外的模型行为表现出来,构成了准确能力评估的根本障碍。我们介绍了一种基于向模型权重中注入不同幅度噪声的新型沙袋检测方法。非沙袋模型在增加噪声时表现出可预测的性能下降,但我们证明沙袋模型表现出异常的性能提升,这可能是由于干扰了欠佳机制,而核心能力仍部分保持完好。通过在各种模型架构、大小和沙袋技术上的实验,我们确立了这种独特的响应模式作为可靠、模型无关的信号,用于检测沙袋行为。重要的是,我们发现噪声注入能够在一个模型表现不佳但未被指示的情况下,使Mistral Large 120B完全展现其性能。我们的发现提供了一种实用的AI评估和监管工具,解决了确保前沿AI系统准确能力评估的挑战。
英文摘要:
Capability evaluations play a crucial role in assessing and regulating frontier AI systems. The effectiveness of these evaluations faces a significant challenge: strategic underperformance, or ``sandbagging'', where models deliberately underperform during evaluation. Sandbagging can manifest either through explicit developer intervention or through unintended model behavior, presenting a fundamental obstacle to accurate capability assessment. We introduce a novel sandbagging detection method based on injecting noise of varying magnitudes into model weights. While non-sandbagging models show predictable performance degradation with increasing noise, we demonstrate that sandbagging models exhibit anomalous performance improvements, likely due to disruption of underperformance mechanisms while core capabilities remain partially intact. Through experiments across various model architectures, sizes, and sandbagging techniques, we establish this distinctive response pattern as a reliable, model-agnostic signal for detecting sandbagging behavior. Importantly, we find noise-injection is capable of eliciting the full performance of Mistral Large 120B in a setting where the model underperforms without being instructed to do so. Our findings provide a practical tool for AI evaluation and oversight, addressing a challenge in ensuring accurate capability assessment of frontier AI systems.
Defending against Indirect Prompt Injection by Instruction Detection
Renmin University of China(中国人民大学);Peking University Shenzhen Graduate School(北京大学深圳研究生院);Wuhan University(武汉大学);University of Science and Technology of China(中国科学技术大学);Hong Kong University of Science and Technology(香港科技大学);Sony AI(索尼人工智能);Microsoft Research Asia(微软亚洲研究院)
PRISM: Recovering Instruction Sets from Language Model Activations
Center for Cybersecurity Systems & Networks, Amrita Vishwa Vidyapeetham(阿姆里塔·维什瓦·维迪亚佩瑟姆网络安全系统与网络中心);Microsoft(微软);Ben-Gurion University of the Negev(内盖夫本-古里安大学)
The Hidden Evolution of Disguised Visual Context inside the VLM
Surrey Institute for People-Centred AI, University of Surrey(萨里大学以人为本人工智能研究所);Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey(萨里大学视觉、语音与信号处理中心)