arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00038cs.CYcs.AI
收藏

LSSF: 通过低秩安全子空间融合实现大语言模型的安全对齐

LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion

East China Normal University(华东师范大学);Ant Group(蚂蚁集团)

Guanghao Zhou, Panjia Qiu, Cen Chen, Hongyu Li, Mingyuan Chu, Xin Zhang, Jun Zhou

AI总结:

LSSF通过低秩安全子空间融合技术,有效恢复微调大语言模型的安全对齐,同时对性能影响较小。

摘要:

大语言模型(LLM)的安全机制表现出显著的脆弱性,即使在没有有害内容的数据集上进行微调,也可能削弱其安全能力。同时,现有安全对齐方法主要依赖微调过程,这导致了计算复杂性和资源需求的增加。为了解决这些问题,我们引入了LSSF,一种新颖的安全再对齐框架,具有低秩安全子空间融合特性。我们提出的方法利用LLM中安全信息的低秩特性,通过构建低秩投影矩阵来提取安全向量的主成分。值得注意的是,这个投影矩阵代表了LLM的低秩安全子空间,在微调过程中保持稳定,并且与模型的通用能力相隔离。这些主成分通过线性运算与微调后的LLM结合,有效恢复安全对齐。此外,为了考虑LLM不同层中安全信息编码密度的差异,我们提出了一种新的度量标准,称为安全奇异值熵。该度量标准量化编码密度,并允许为每个安全向量动态计算安全关键秩。大量实验表明,我们提出的方法可以有效恢复微调模型的安全对齐,同时对下游任务的性能影响最小。

英文摘要:

The safety mechanisms of large language models (LLMs) exhibit notable fragility, as even fine-tuning on datasets without harmful content may still undermine their safety capabilities. Meanwhile, existing safety alignment methods predominantly rely on the fine-tuning process, which inadvertently leads to the increased complexity and computational resources required. To address these issues, we introduce LSSF, a novel safety re-alignment framework with \underline{L}ow-Rank \underline{S}afety \underline{S}ubspace \underline{F}usion. Our proposed method exploits the low-rank characteristics of safety information in LLMs by constructing a low-rank projection matrix to extract the principal components of safety vectors. Notably, this projection matrix represents the low-rank safety subspace of the LLMs, which we have observed to remain stable during fine-tuning process and is isolated from the model's general capabilities. These principal components are used to effectively restore safety alignment when combined with fine-tuned LLMs through linear arithmetic. Additionally, to account for the varying encoding densities of safety information across different layers of LLMs, we propose a novel metric called safety singular value entropy. This metric quantifies the encoding density and allows for the dynamic computation of the safety-critical rank for each safety vector. Extensive experiments demonstrate that our proposed post-hoc alignment method can effectively restore the safety alignment of fine-tuned models with minimal impact on their performance in downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑