通过推理时间激活能量缓解对齐大语言模型中的过度拒绝
Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energy
AI总结：
通过能量景观引导框架，提升大语言模型的安全性并减少虚假拒绝，实验显示在ORB-H基准上合规性显著提高。
摘要：
大语言模型的安全对齐目前面临一个核心挑战：现有的对齐技术往往优先缓解对有害提示的响应，而牺牲了过度谨慎的行为，导致模型错误拒绝良性请求。因此，安全对齐的核心目标是在提高安全性的同时最小化虚假拒绝。在本工作中，我们引入了能量景观引导（ELS），一种新的、无需微调的框架，旨在通过动态的推理时间干预解决这一挑战。我们训练了一个轻量级的外部能量基模型（EBM），以将高能量分配给不希望的状态（虚假拒绝或劫持）并低能量分配给希望的状态（有益响应或安全拒绝）。在推理过程中，EBM将LLM的内部激活映射到一个能量景观，并利用能量函数的梯度实时引导隐藏状态向低能量区域。这种动态引导模型向期望行为，而无需修改其参数。通过解耦行为控制与模型的核心知识，ELS提供了一种灵活且计算高效的解决方案。在多样化的模型上进行的广泛实验展示了其有效性，使ORB-H基准的合规性从57.3%提升至82.6%，同时保持基线安全性能。我们的工作建立了一种有前景的范式，用于构建同时实现高安全性和低虚假拒绝率的LLM。
英文摘要：
Safety alignment of large language models currently faces a central challenge: existing alignment techniques often prioritize mitigating responses to harmful prompts at the expense of overcautious behavior, leading models to incorrectly refuse benign requests. A key goal of safe alignment is therefore to improve safety while simultaneously minimizing false refusals. In this work, we introduce Energy Landscape Steering (ELS), a novel, fine-tuning free framework designed to resolve this challenge through dynamic, inference-time intervention. We train a lightweight external Energy-Based Model (EBM) to assign high energy to undesirable states (false refusal or jailbreak) and low energy to desirable states (helpful response or safe reject). During inference, the EBM maps the LLM's internal activations to an energy landscape, and we use the gradient of the energy function to steer the hidden states toward low-energy regions in real time. This dynamically guides the model toward desirable behavior without modifying its parameters. By decoupling behavioral control from the model's core knowledge, ELS provides a flexible and computationally efficient solution. Extensive experiments across diverse models demonstrate its effectiveness, raising compliance on the ORB-H benchmark from 57.3 percent to 82.6 percent while maintaining baseline safety performance. Our work establishes a promising paradigm for building LLMs that simultaneously achieve high safety and low false refusal rates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
