通过自然语言反馈学习实现个性化问答
Learning from Natural Language Feedback for Personalized Question Answering
AI总结：
本文提出VAC框架，利用自然语言反馈替代标量奖励，提升个性化问答效果，实验表明其在LaMP-QA基准上表现优异。
摘要：
个性化对增强语言技术的有效性和用户满意度至关重要，特别是在信息检索任务如问答中。当前基于大型语言模型（LLM）的个性化方法通常依赖检索增强生成（RAG），随后通过标量奖励信号进行强化学习。我们发现这些标量奖励有时提供弱且非指导性的反馈，限制了学习效率和个性化质量。我们引入VAC，一种新的个性化响应生成框架，用基于用户资料和问题叙述生成的自然语言反馈（NLF）替代标量奖励。NLF作为丰富且可操作的监督信号，使策略模型能够迭代优化其输出并内化有效的个性化策略。训练过程交替优化反馈模型和在改进响应上微调策略模型，最终使策略模型在推理时不再需要反馈。在由三个多样化领域组成的LaMP-QA基准上的评估显示，VAC在状态-of-the-art结果上取得了一致且显著的改进。人类评估进一步确认了生成响应的优越质量。这些结果表明，NLF为优化个性化问答提供了更有效的信号。
英文摘要：
Personalization is crucial for enhancing both the effectiveness and user satisfaction of language technologies, particularly in information-seeking tasks like question answering. Current approaches for personalizing large language models (LLMs) often rely on retrieval-augmented generation (RAG), followed by reinforcement learning with scalar reward signals to teach models how to use retrieved personal context. We believe that these scalar rewards sometimes provide weak, non-instructive feedback, limiting learning efficiency and personalization quality. We introduce VAC, a novel framework for personalized response generation that replaces scalar rewards with natural language feedback (NLF) that are generated conditioned on the user profiles and the question narratives. NLF serves as a rich and actionable supervision signal, allowing the policy model to iteratively refine its outputs and internalize effective personalization strategies. Training alternates between optimizing the feedback model and fine-tuning the policy model on the improved responses, resulting in a policy model that no longer requires feedback at inference. Evaluation on the LaMP-QA benchmark that consists of three diverse domains demonstrates consistent and significant improvements over the state-of-the-art results. Human evaluations further confirm the superior quality of the generated responses. These results demonstrate that NLF provides more effective signals for optimizing personalized question answering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
