我会对你撒谎吗？使用直接偏好头在推理时对齐语言模型
Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
AI总结：
本文提出直接偏好头（DPH）微调框架，通过辅助奖励头学习人类偏好信号而不直接影响语言建模头的输出分布，解决RLHF损害推理能力和引入幻觉的问题，在GLUE、RACE和GPT4All上表现优于单独使用SFT或DPO。
摘要：
预训练语言模型（LMs）展现出强大的零样本和上下文学习能力；然而，它们的行为往往难以控制。通过利用人类反馈强化学习（RLHF），可以对无监督LMs进行微调，使其遵循指令并生成反映人类偏好的输出。尽管具有这些优势，但RLHF已被证明可能会损害语言模型的推理能力，并引入诸如幻觉等伪影，其中模型可能会捏造事实。为了解决这个问题，我们引入了直接偏好头（DPH），这是一种微调框架，使LMs能够通过辅助奖励头学习人类偏好信号，而不会直接影响语言建模头的输出分布。我们对目标函数进行了理论分析，并发现其与保守直接偏好优化（cDPO）存在紧密联系。最后，我们在GLUE、RACE和GPT4All评估套件上评估了我们的模型，并证明我们的方法生成的模型比仅使用监督微调（SFT）或直接偏好优化（DPO）微调的模型获得更高的分数。
英文摘要：
Pre-trained Language Models (LMs) exhibit strong zero-shot and in-context learning capabilities; however, their behaviors are often difficult to control. By utilizing Reinforcement Learning from Human Feedback (RLHF), it is possible to fine-tune unsupervised LMs to follow instructions and produce outputs that reflect human preferences. Despite its benefits, RLHF has been shown to potentially harm a language model's reasoning capabilities and introduce artifacts such as hallucinations where the model may fabricate facts. To address this issue we introduce Direct Preference Heads (DPH), a fine-tuning framework that enables LMs to learn human preference signals through an auxiliary reward head without directly affecting the output distribution of the language modeling head. We perform a theoretical analysis of our objective function and find strong ties to Conservative Direct Preference Optimization (cDPO). Finally we evaluate our models on GLUE, RACE, and the GPT4All evaluation suite and demonstrate that our method produces models which achieve higher scores than those fine-tuned with Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO) alone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
