arXiv 2502.086572025-02-08cs.CLcs.AI

通过精炼正向和有毒样本实现LLM的双重安全自对齐，且人类干预最小化

Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions

National Engineering Research Center for Mobile Network Technologies, Beijing University of Posts and Telecommunications（北京邮电大学国家移动网络技术工程研究中心）；Nanyang Technological University（南洋理工大学）；Guangzhou University（广州大学）

Jingxin Xu, Guoshun Nan, Sheng Guan, Sicong Leng, Yilian Liu, Zixiao Wang, Yuyang Ma, Zhili Zhou, Yanzhao Hou, Xiaofeng Tao

备注：: This work has been submitted to the IEEE for possible publication

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对LLMs安全对齐依赖人工标注、丢失有毒样本负向参考的问题，提出PT-ALIGN方法，通过自动精炼正向与有毒样本、仅需少于50条人类标注，实现双重安全自对齐，在9个开源LLMs上验证了有效性且保持有用性。

摘要：

近期的AI智能体（如ChatGPT和LLaMA）主要依赖指令微调与强化学习，使大型语言模型（LLMs）的输出符合人类意图，确保其无害且有用。现有方法严重依赖高质量正向样本的人工标注，同时面临噪声标签、偏好与非偏好响应数据区分度低等问题。然而，具有明确安全区分度的易获取有毒样本常被过滤，丢失了可辅助LLMs安全对齐的宝贵负向参考。为此，我们提出PT-ALIGN——一种新型安全自对齐方法，通过自动精炼正向与有毒样本并执行细粒度双重指令微调，最小化人类监督。正向样本为无害响应，有毒样本则刻意包含极有害内容，作为新的监督信号。具体而言，我们仅利用少于50条人类标注，让LLM自身迭代生成并精炼训练实例；随后采用最大似然估计（MLE）与细粒度非似然训练（UT）两种损失函数联合学习，以增强LLMs的安全性：MLE损失鼓励LLM基于正向样本最大化无害内容生成，细粒度UT损失则引导LLM在token层面基于负向样本最小化有害词汇输出，从而实现安全与有效性的解耦，指向更安全的微调目标，并提升生成内容的有用性与可靠性。在9个主流开源LLMs上的实验表明，PT-ALIGN在安全对齐方面有效，同时保持了相当的有用性水平。

英文摘要：

Recent AI agents, such as ChatGPT and LLaMA, primarily rely on instruction tuning and reinforcement learning to calibrate the output of large language models (LLMs) with human intentions, ensuring the outputs are harmless and helpful. Existing methods heavily depend on the manual annotation of high-quality positive samples, while contending with issues such as noisy labels and minimal distinctions between preferred and dispreferred response data. However, readily available toxic samples with clear safety distinctions are often filtered out, removing valuable negative references that could aid LLMs in safety alignment. In response, we propose PT-ALIGN, a novel safety self-alignment approach that minimizes human supervision by automatically refining positive and toxic samples and performing fine-grained dual instruction tuning. Positive samples are harmless responses, while toxic samples deliberately contain extremely harmful content, serving as a new supervisory signals. Specifically, we utilize LLM itself to iteratively generate and refine training instances by only exploring fewer than 50 human annotations. We then employ two losses, i.e., maximum likelihood estimation (MLE) and fine-grained unlikelihood training (UT), to jointly learn to enhance the LLM's safety. The MLE loss encourages an LLM to maximize the generation of harmless content based on positive samples. Conversely, the fine-grained UT loss guides the LLM to minimize the output of harmful words based on negative samples at the token-level, thereby guiding the model to decouple safety from effectiveness, directing it toward safer fine-tuning objectives, and increasing the likelihood of generating helpful and reliable content. Experiments on 9 popular open-source LLMs demonstrate the effectiveness of our PT-ALIGN for safety alignment, while maintaining comparable levels of helpfulness and usefulness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。