Compliance2LoRA：通过超网络生成的LoRA适配器对任意策略子集进行按需安全对齐

中文摘要 AI 辅助

大型推理模型（LRM）的后训练对齐已显著提升其对不同安全合规场景的适应性。但随着LRM针对下游用户的个性化成为核心，对不同合规级别的需求不断增长——不同用户特定的LRM需遵循不同的安全策略子集。为每个策略子集单独训练LRM会带来严重的组合开销。虽然上下文学习方法可克服该组合开销，但会引入与长上下文生成相关的额外计算挑战。为应对这一挑战，我们提出了Compliance2LoRA（即ours），一种用于多策略合规的统一自适应超网络框架。在该框架中，安全策略作为可定制输入传入LoRA适配器生成器，该生成器学习为下游LRM生成符合策略的LoRA权重。将这些权重添加到LRM后，可生成符合指定策略子集的响应。本研究表明，训练此类超网络可在单个LRM上实现按需策略调整，且不会在不同规模推理模型和不同评估数据集上牺牲任务性能，凸显了基于自适应超网络的对齐在LRM中的有效性与实用性。

英文摘要

Post-training alignment in large reasoning models (LRMs) has significantly improved their adaptability to diverse safety compliance settings. However, as LRMs personalization for downstream users takes center stage, the demand for varying levels of policy compliance grows as different user-specific LRMs must adhere to distinct subsets of safety policies. Training a separate LRM for each policy subset introduces severe combinatorial overhead. While in context learning methods overcome this combinatorial overhead, they introduce additional computational challenges associated with long context generation. To address this challenge, we propose \ours, a unified adaptive hypernetwork-based framework for multi-policy compliance. In our framework, safety policies serve as customizable inputs to a LoRA adapter generator, which learns to produce policy compliant LoRA weights for downstream LRM. When added to the LRM these weights enable the generation of responses compliant with the specified policy subsets. In this work, we demonstrate that training such a hypernetwork enables on-demand policy adjustments on a single LRM without sacrificing task performance across reasoning models of different sized and different evaluation datasets. This highlights the effectiveness and practicality of adaptive hypernetwork based alignment in LRMs.

发表机构

• University of Maryland College Park（马里兰大学帕克分校）

机构由 AI 辅助整理，请以论文原文为准。