WildFeedback: 对齐大语言模型与实时用户交互与反馈
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
AI总结：
本文提出WildFeedback框架，通过实时用户反馈自动创建偏好数据集，提升大语言模型对用户偏好的对齐能力，解决了现有方法的可扩展性、主观性和偏见问题。
摘要：
随着大语言模型（LLMs）的持续发展，将这些模型与人类偏好对齐已成为关键挑战。传统对齐方法依赖于人工或LLM标注的数据集，受限于资源密集型、固有主观性、与真实用户偏好不匹配以及反馈循环放大模型偏见的风险。为克服这些限制，我们引入WildFeedback，一种新的框架，利用与LLM对话时的实时用户反馈来自动创建偏好数据集。给定一个多轮用户-LLM对话语料，WildFeedback识别并分类用户反馈到LLM响应之间对话轮次。然后将用户反馈用于根据用户偏好创建首选和不首选的响应示例。我们的实验表明，使用WildFeedback数据集微调的LLM在对齐用户偏好方面显著提高，这在传统基准和我们提出的检查表引导评估中得到证实。通过整合实际用户的实时反馈，WildFeedback解决了现有方法的可扩展性、主观性和偏见问题，标志着向开发更响应用户多样化和不断变化需求的LLM迈出重要一步。
英文摘要：
As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, misalignment with real-world user preferences, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages in-situ user feedback during conversations with LLMs to create preference datasets automatically. Given a corpus of multi-turn user-LLM conversation, WildFeedback identifies and classifies user feedback to LLM responses between conversation turns. The user feedback is then used to create examples of preferred and dispreferred responses according to users' preference. Our experiments demonstrate that LLMs fine-tuned on WildFeedback dataset exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed checklist-guided evaluation. By incorporating in-situ feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
