从负反馈、正反馈或两者中学习
Learning from negative feedback, or positive feedback or both
AI总结：
本文提出一种解耦正负反馈学习的EM扩展方法，支持仅用单一类型反馈进行稳定学习，并验证了其在语言模型人类反馈训练和序贯决策中的有效性。
摘要：
现有的偏好优化方法通常假设存在成对偏好反馈（偏好/正例与不偏好/负例）的场景。这一要求限制了它们在仅有非成对反馈——例如只有正反馈或只有负反馈——的场景中的适用性。为了解决这一问题，我们提出了一种新方法，将正反馈学习和负反馈学习解耦。这种解耦使得能够控制每种反馈类型的影响，并且重要的是，即使只有一种反馈类型存在，也能进行学习。一个关键贡献是证明了仅从负反馈中进行稳定学习的能力，这是当前方法未能很好解决的问题。我们的方法建立在（Dayan and Hinton, 1997）中引入的概率框架之上，该框架使用期望最大化（EM）直接优化正结果的概率（与经典的期望奖励最大化相对）。我们解决了当前基于EM方法的一个关键局限：它们仅最大化正例的似然，而忽略了负例。我们展示了如何扩展EM算法以显式纳入负例，从而得到一个具有理论基础的算法，为从正反馈和负反馈中学习提供了一种直观且通用的方式。我们评估了该方法在基于人类反馈训练语言模型以及训练序贯决策问题策略（其中学习到的价值函数可用）中的表现。
英文摘要：
Existing preference optimization methods often assume scenarios where paired preference feedback (preferred/positive vs. dis-preferred/negative examples) is available. This requirement limits their applicability in scenarios where only unpaired feedback--for example, either positive or negative--is available. To address this, we introduce a novel approach that decouples learning from positive and negative feedback. This decoupling enables control over the influence of each feedback type and, importantly, allows learning even when only one feedback type is present. A key contribution is demonstrating stable learning from negative feedback alone, a capability not well-addressed by current methods. Our approach builds upon the probabilistic framework introduced in (Dayan and Hinton, 1997), which uses expectation-maximization (EM) to directly optimize the probability of positive outcomes (as opposed to classic expected reward maximization). We address a key limitation in current EM-based methods: they solely maximize the likelihood of positive examples, while neglecting negative ones. We show how to extend EM algorithms to explicitly incorporate negative examples, leading to a theoretically grounded algorithm that offers an intuitive and versatile way to learn from both positive and negative feedback. We evaluate our approach for training language models based on human feedback as well as training policies for sequential decision-making problems, where learned value functions are available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
