对齐到什么？基于RLHF的对齐的局限性
Aligning to What? Limits to RLHF Based Alignment
AI总结：
本文研究RLHF对齐技术在减轻LLMs隐性偏见上的局限，发现SFT会固化偏见，并指出当前对齐方法不足以处理模糊的偏见缓解任务。
摘要：
基于人类反馈的强化学习（RLHF）越来越多地被用于使大型语言模型（LLMs）与人类偏好对齐。然而，RLHF在解决潜在偏见方面的有效性仍不清楚。本研究考察了RLHF与LLMs中隐性偏见和显性偏见之间的关系，特别关注针对非裔美国人的偏见。我们将多种RLHF技术（DPO、ORPO和RLOO）应用于Llama 3 8B，并使用匹配伪装探测和显式偏见测试评估所得模型的隐性偏见和显性偏见。我们在不同基础模型和数据集上使用DPO进行了额外测试；在若干启示中，我们发现RLHF之前的SFT会使模型偏见固化。此外，我们将测量偏见的工具扩展到多模态模型。通过实验，我们收集到的证据表明，当前的对齐技术不足以应对诸如减轻隐性偏见这类模糊任务，这凸显了对高质量数据集、数据整理技术或对齐工具的需求。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) is increasingly used to align large language models (LLMs) with human preferences. However, the effectiveness of RLHF in addressing underlying biases remains unclear. This study investigates the relationship between RLHF and both covert and overt biases in LLMs, particularly focusing on biases against African Americans. We applied various RLHF techniques (DPO, ORPO, and RLOO) to Llama 3 8B and evaluated the covert and overt biases of the resulting models using matched-guise probing and explicit bias testing. We performed additional tests with DPO on different base models and datasets; among several implications, we found that SFT before RLHF calcifies model biases. Additionally, we extend the tools for measuring biases to multi-modal models. Through our experiments we collect evidence that indicates that current alignment techniques are inadequate for nebulous tasks such as mitigating covert biases, highlighting the need for capable datasets, data curating techniques, or alignment tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
