超越奖励黑客：用于大型语言模型对齐的因果奖励
Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
AI总结：
本文提出一种新颖的因果奖励建模方法，通过整合因果性并强制反事实不变性来缓解RLHF中的虚假相关性，有效消除多种偏差，从而实现更可靠、公平的大型语言模型对齐。
摘要：
大型语言模型（LLMs）的最新进展展示了在执行复杂任务方面的显著进步。尽管基于人类反馈的强化学习（RLHF）在使LLMs与人类偏好对齐方面非常有效，但它容易受到奖励建模中虚假相关性的影响。因此，它经常引入偏差——如长度偏差、谄媚、概念偏差和歧视——这些偏差阻碍了模型捕捉真实因果关系的能力。为了解决这个问题，我们提出了一种新颖的因果奖励建模方法，该方法整合因果性来缓解这些虚假相关性。我们的方法强制执行反事实不变性，确保在改变无关变量时奖励预测保持一致。通过对合成和真实世界数据集的实验，我们表明我们的方法有效缓解了各种类型的虚假相关性，从而使LLMs与人类偏好的对齐更加可靠和公平。作为现有RLHF工作流程的即插即用增强，我们的因果奖励建模为提高LLM微调的可信度和公平性提供了一种实用的方式。
英文摘要：
Recent advances in large language models (LLMs) have demonstrated significant progress in performing complex tasks. While Reinforcement Learning from Human Feedback (RLHF) has been effective in aligning LLMs with human preferences, it is susceptible to spurious correlations in reward modeling. Consequently, it often introduces biases-such as length bias, sycophancy, conceptual bias, and discrimination-that hinder the model's ability to capture true causal relationships. To address this, we propose a novel causal reward modeling approach that integrates causality to mitigate these spurious correlations. Our method enforces counterfactual invariance, ensuring reward predictions remain consistent when irrelevant variables are altered. Through experiments on both synthetic and real-world datasets, we show that our approach mitigates various types of spurious correlations effectively, resulting in more reliable and fair alignment of LLMs with human preferences. As a drop-in enhancement to the existing RLHF workflow, our causal reward modeling provides a practical way to improve the trustworthiness and fairness of LLM finetuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
