对幻觉的对齐：人类和AI反馈中的选择盲区
Aligning to Illusions: Choice Blindness in Human and AI Feedback
AI总结：
研究揭示RLHF中偏好构建的问题，显示人类和AI反馈中的选择盲区受上下文影响，导致奖励信号失真。
摘要：
强化学习从人类反馈（RLHF）假设标注者偏好反映稳定的内部状态。我们通过三个实验跨越偏好流程来挑战这一假设。在一项人类选择盲区研究中，91%的隐秘交换偏好未被检测到，将选择盲区扩展到对不熟悉文本的第三人评价比较。测试十五个LLM法官作为潜在替代者，我们发现检测依赖于浅层文本匹配而不是真正的自我监控：移除上下文中的先前推理会导致盲区从接近零上升到超过50%，而显性的社会压力则导致几乎普遍的服从。在一项剂量反应实验中，跨两种架构（从86M到2B参数）的两个模型中，每六到三分之一的标签被破坏后，奖励信号才会减半，但标准成对准确率几乎保持不变。最佳N评估证实了这一点转化为下游策略退化：在50%的破坏情况下，奖励引导选择无法比随机采样有所改进，而代理模型报告的分数单调递增。这些结果揭示了一个偏好构建问题：进入RLHF的信号受 elicitation 上下文 形成的方式，这种方式无法被人类元认知、LLM自我监控或标准评估指标所检测。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) assumes annotator preferences reflect stable internal states. We challenge this through three experiments spanning the preference pipeline. In a human choice blindness study, 91% of surreptitiously swapped preferences go undetected, extending choice blindness to third-person evaluative comparison of unfamiliar text. Testing fifteen LLM judges as potential replacements, we find detection relies on shallow text matching rather than genuine self-monitoring: removing prior reasoning from context causes blindness to surge from near-zero to over 50%, while explicit social pressure induces near-universal compliance. In a dose-response experiment across two architectures from 86M to 2B parameters, one-sixth to one-third of labels must be corrupted before the reward signal halves, yet standard pairwise accuracy remains virtually unchanged. A Best-of-N evaluation confirms this translates to downstream policy degradation: at 50% corruption, reward-guided selection produces no improvement over random sampling, while the proxy model reports monotonically increasing scores. Together, these results reveal a preference construction problem: the signal entering RLHF is shaped by elicitation context in ways that neither human metacognition, LLM self-monitoring, nor standard evaluation metrics can detect.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
