InferAligner：通过跨模型引导实现推理时无害性对齐
InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance
AI总结：
本文提出InferAligner，通过安全对齐模型提取安全引导向量，在推理时修改目标模型激活以实现无害性对齐，可降低有害指令与越狱攻击ASR且几乎不影响下游性能。
摘要：
随着大型语言模型（LLM）的快速发展，它们不仅被用作通用AI助手，还通过进一步微调进行定制，以满足不同应用的要求。当前LLM成功的一个关键因素是对齐过程。当前的对齐方法，如监督微调（SFT）和基于人类反馈的强化学习（RLHF），侧重于训练时对齐，实施起来往往复杂且繁琐。因此，我们开发了InferAligner，这是一种新颖的推理时对齐方法，利用跨模型引导进行无害性对齐。InferAligner利用从安全对齐模型中提取的安全引导向量，在目标模型响应有害输入时修改其激活，从而引导目标模型提供无害响应。实验结果表明，我们的方法可以非常有效地应用于金融、医学和数学领域的领域专用模型，以及LLaVA等多模态大型语言模型（MLLM）。它显著降低了有害指令和越狱攻击的攻击成功率（ASR），同时保持下游任务性能几乎不变。
英文摘要：
With the rapid development of large language models (LLMs), they are not only used as general-purpose AI assistants but are also customized through further fine-tuning to meet the requirements of different applications. A pivotal factor in the success of current LLMs is the alignment process. Current alignment methods, such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), focus on training-time alignment and are often complex and cumbersome to implement. Therefore, we develop \textbf{InferAligner}, a novel inference-time alignment method that utilizes cross-model guidance for harmlessness alignment. InferAligner utilizes safety steering vectors extracted from safety-aligned model to modify the activations of the target model when responding to harmful inputs, thereby guiding the target model to provide harmless responses. Experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics, as well as to multimodal large language models (MLLMs) such as LLaVA. It significantly diminishes the Attack Success Rate (ASR) of both harmful instructions and jailbreak attacks, while maintaining almost unchanged performance in downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
