社会强化学习中的目标解耦：从趋炎附势的多数中恢复真实目标
Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majorities
AI总结：
本文提出Epistemic Source Alignment（ESA）方法，解决社会强化学习中因多数评估者偏见导致的目标解耦问题，通过判断反馈来源而非信号本身，确保收敛到真实目标。
摘要：
当代人工智能对齐策略依赖于一个脆弱的前提：即人类反馈虽然嘈杂，但仍是一种根本真实的信号。在本文中，我们将这一假设视为强化学习（RL）中的第4条教条。我们证明，尽管在静态环境中这一教条成立，但在社会环境中，评估者可能具有趋炎附势、懒惰或敌对的倾向时，这一教条会失效。我们证明，在第4条教条下，标准RL智能体会遭遇我们称为目标解耦的结构性故障模式，即智能体学习到的目标会永久性地与潜在的真实目标分离，从而保证收敛到不一致的状态。为了解决这一问题，我们提出了认知源对齐（ESA）。与标准稳健方法依赖统计共识（信任多数）不同，ESA利用稀疏安全公理来判断反馈的来源，而不是信号本身。我们证明，这种
英文摘要：
Contemporary AI alignment strategies rely on a fragile premise: that human feedback, while noisy, remains a fundamentally truthful signal. In this paper, we identify this assumption as Dogma 4 of Reinforcement Learning (RL). We demonstrate that while this dogma holds in static environments, it fails in social settings where evaluators may be sycophantic, lazy, or adversarial. We prove that under Dogma 4, standard RL agents suffer from what we call Objective Decoupling, a structural failure mode where the agent's learned objective permanently separates from the latent ground truth, guaranteeing convergence to misalignment. To resolve this, we propose Epistemic Source Alignment (ESA). Unlike standard robust methods that rely on statistical consensus (trusting the majority), ESA utilizes sparse safety axioms to judge the source of the feedback rather than the signal itself. We prove that this "judging the judges" mechanism guarantees convergence to the true objective, even when a majority of evaluators are biased. Empirically, we show that while traditional consensus methods fail under majority collusion, our approach successfully recovers the optimal policy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
