面向生成式语言模型的负向提示驱动对齐
Negative-Prompt-driven Alignment for Generative Language Model
AI总结：
针对现有对齐方法忽视负例的问题，提出 NEAT 框架，通过引入负向提示和双重反馈机制，利用扩展偏好数据集的排序损失进行在线对齐，显著提升语言模型与人类价值观和偏好的对齐效果。
摘要：
大型语言模型已展现出卓越的能力，但将其输出与人类价值观和偏好相对齐仍是一项重大挑战。现有的对齐方法主要关注正例，却忽视了负向响应在引导模型远离不良行为方面的重要性。例如，广泛使用的对齐数据集显示，缺乏与人类价值观相悖的显式负例，这阻碍了其在训练过程中抑制有害或带有偏见输出的能力。为了解决这一局限性，我们提出了 NEAT（即负向提示驱动对齐），在优化过程中引入负向提示，与正例一起生成不良响应。NEAT 对产生有害输出的模型进行显式惩罚，不仅引导其趋向期望行为，还使其避免生成不良、偏见的响应。这种双重反馈机制实现了与人类偏好更好的对齐，在避免伤害至关重要的场景中尤为关键。从一个预训练语言模型出发，NEAT 通过引入基于包含正负例的扩展偏好数据集的排序损失，执行在线对齐。大量实验验证了 NEAT 在显著增强语言模型与人类价值观和偏好对齐方面的有效性。
英文摘要：
Large language models have achieved remarkable capabilities, but aligning their outputs with human values and preferences remains a significant challenge. Existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors. For instance, the widely-used alignment datasets reveals a scarcity of explicit negative examples that contradict human values, hindering its ability to discourage harmful or biased outputs during training. To address this limitation, we propose NEAT, i.e., NEgative-prompt-driven AlignmenT, to introduce negative prompts to generate undesirable responses alongside positive examples during the optimization process. NEAT explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. This dual feedback mechanism enables better alignment with human preferences, crucial in contexts where avoiding harm is paramount. Starting from a pre-trained language model, NEAT performs online alignment by incorporating a ranking loss derived from an expanded preference dataset containing both positive and negative examples. Extensive experiments validate NEAT's effectiveness in significantly enhancing language models' alignment with human values and preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
