发表机构
Stanford University; Amazon; University of Washington（斯坦福大学; 亚马逊; 华盛顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出多元偏好优化（PlurPO）方法，通过让语言模型模拟多方利益相关者视角来减少社交谄媚，在四个数据集和四个模型上显著降低认可率，并实现跨模型迁移。
AI 中文摘要
个人建议（包括关系建议）现在已成为生成式人工智能最常见的用途之一。然而，语言模型（LMs）表现出谄媚行为：它们比人类更频繁地肯定用户，这可能会使人们过度自信，并在冲突后更不愿意修复他们的关系。先前关于缓解谄媚行为的工作集中在事实性场景中，在这些场景中，响应可以对照真实答案进行检查，而针对社交谄媚行为（例如个人建议，其中没有真实答案）的缓解措施则依赖于简单的提示和训练后方法，其效果有限。我们的洞察是，社交谄媚行为之所以发生，部分原因是语言模型过度以用户为中心，未能考虑受用户行为影响的其他利益相关者的视角。为解决这一问题，我们提出了多元偏好优化（PlurPO）：给定描述人际冲突的输入，语言模型识别并模拟相关利益相关者，然后训练其偏好并生成所有利益相关者都能接受的响应。PlurPO仅使用模型自身对其输出产生的信号，无需真实标签。与先前方法相比，PlurPO在四个数据集和四个模型家族上显著减少了社交谄媚行为。例如，在意图造成伤害的陈述中（用户的行动不应被认可），PlurPO在四个模型上将认可率平均降低了89%。在一般建议问题上（目标是匹配人类响应的认可率），它将差距缩小了一半以上，平均从17.8%降至8.0%。PlurPO为8B模型构建的偏好数据集也能有效迁移到更大的（32B）模型中，以缓解谄媚行为。我们的结果表明，通过利用模型自身的能力来模拟多种相关视角，可以减少社交谄媚行为。
英文摘要
Personal advice, including relationship advice, now ranks among the most common uses of generative AI. But language models (LMs) exhibit sycophancy: they affirm users much more often than humans do, which can make people overconfident and less willing to repair their relationships after a conflict. Prior work on mitigating sycophancy has focused on factual settings where a response can be checked against a ground truth answer, while mitigations for social sycophancy (e.g., personal advice, where there is no ground truth) have relied on simple prompting and post-training methods with limited effectiveness. Our insight is that social sycophancy occurs in part because LMs overly center on the user and fail to consider the perspectives of other stakeholders impacted by the user's behavior. To address this problem we propose Pluralistic Preference Optimization (PlurPO): given inputs describing interpersonal conflicts, the LM identifies and simulates the relevant stakeholders, and is then trained to prefer and generate responses acceptable to all stakeholders. PlurPO uses only signals the model produces about its own outputs, without ground-truth labels. PlurPO substantially reduces social sycophancy across four datasets and four model families compared to prior methods. For example, on statements of intent to cause harm, where the users' actions should not be endorsed, PlurPO reduces the endorsement rate by 89% on average across four models. On general advice questions, where the target is to match the endorsement rate of human responses, it closes the gap by more than half, from 17.8% to 8.0% on average. The preference dataset constructed by PlurPO for an 8B model also effectively transfers to mitigating sycophancy in a larger (32B) model. Our results indicate that social sycophancy can be reduced by leveraging a model's own capabilities to simulate a plurality of relevant perspectives.
