arXiv 2507.042502025-07-06cs.LGcs.AI

恰到好处的转变：通过定向表示微调缓解对齐语言模型中的过度拒绝

Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning

Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA, USA（弗吉尼亚理工学院与州立大学电气与计算机工程系）；Cisco Research, San Jose, CA, USA（思科研究）

Mahavir Dabas, Si Chen, Charles Fleming, Ming Jin, Ruoxi Jia

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对对齐语言模型的过度拒绝问题，提出ACTOR框架，利用内部激活模式并仅微调单层模型，在保持安全性和实用性的同时有效减少对良性提示的拒绝。

摘要：

安全对齐对于大语言模型（LLMs）抵御恶意指令至关重要，但往往会导致过度拒绝，即良性提示被不必要地拒绝，从而损害用户体验和模型实用性。我们提出ACTOR（Activation-Based Training for Over-Refusal Reduction，基于激活的训练以减少过度拒绝），这是一个稳健且计算和数据高效的训练框架，通过利用来自多样化查询的内部激活模式来最小化过度拒绝。ACTOR精确识别并调整触发拒绝的激活成分，从而对拒绝机制提供更强的控制。通过仅微调单个模型层，ACTOR在多个基准上有效减少了过度拒绝，同时保持了模型处理有害查询的能力并保留了整体实用性。

英文摘要：

Safety alignment is crucial for large language models (LLMs) to resist malicious instructions but often results in over-refusals, where benign prompts are unnecessarily rejected, impairing user experience and model utility. We introduce ACTOR (Activation-Based Training for Over-Refusal Reduction), a robust and compute- and data-efficient training framework that minimizes over-refusals by leveraging internal activation patterns from diverse queries. ACTOR precisely identifies and adjusts the activation components that trigger refusals, providing stronger control over the refusal mechanism. By fine-tuning only a single model layer, ACTOR effectively reduces over-refusals across multiple benchmarks while maintaining the model's ability to handle harmful queries and preserve overall utility.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。