使用反事实 DPO 对齐大型语言模型
Aligning Large Language Models with Counterfactual DPO
AI总结：
本文提出在 DPO 框架中使用反事实提示，无需人工干预即可对齐 LLM 风格、强化良好行为并拒绝不当指令，实现低资源且符合伦理的微调。
摘要：
大型语言模型（LLM）的进步已在各类应用中展现出卓越能力。这些模型擅长生成上下文连贯且涵盖广泛主题的文本补全。然而，训练它们所需的海量数据集使预训练和指令微调阶段中的回复风格对齐颇具挑战。因此，通常会采用额外的对齐阶段，在该阶段使用人类偏好数据进一步训练模型，使其输出更好地符合人类期望。虽然这一过程本身并不会引入新能力，但它确实会强化模型固有的生成风格。本文探讨在 Direct Preference Optimization（DPO，直接偏好优化）框架内利用反事实提示，在不依赖人工干预的情况下对齐模型风格。我们证明，该方法能有效灌输可取行为、抑制不可取行为，并促使模型忽略不当指令。研究结果表明，结合 DPO 的反事实提示提供了一种低资源方式来微调 LLM，以满足对负责任且符合伦理的 AI 系统的需求。
英文摘要：
Advancements in large language models (LLMs) have demonstrated remarkable capabilities across a diverse range of applications. These models excel in generating text completions that are contextually coherent and cover an extensive array of subjects. However, the vast datasets required for their training make aligning response styles during the pretraining and instruction tuning phases challenging. Consequently, an additional alignment phase is typically employed, wherein the model is further trained with human preference data to better align its outputs with human expectations. While this process doesn't introduce new capabilities per se, it does accentuate generation styles innate to the model. This paper explores the utilization of counterfactual prompting within the framework of Direct Preference Optimization (DPO) to align the model's style without relying on human intervention. We demonstrate that this method effectively instils desirable behaviour, mitigates undesirable ones, and encourages the model to disregard inappropriate instructions. Our findings suggest that counterfactual prompting with DPO presents a low-resource way to fine-tune LLMs to meet the demands for responsible and ethically aligned AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
