教导语言模型与用户共同进化：面向个性化对齐的动态资料模型
Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
AI总结：
本研究提出RLPA框架，通过动态资料推断提升个性化对话性能，Qwen-RLPA在多个基准测试中超越现有方法。
摘要：
个性化对齐对于使大型语言模型（LLMs）有效地参与以用户为中心的对话至关重要。尽管最近的基于提示和离线优化方法提供了初步解决方案，但它们在冷启动场景和长期个性化方面存在不足，因为其本质上是静态和浅层的。在本工作中，我们引入了个性化对齐强化学习（RLPA）框架，其中LLM与模拟用户模型互动，通过对话迭代推断和优化用户资料。训练过程由双层奖励结构引导：资料奖励鼓励准确构建用户表示，而响应奖励激励生成与推断资料一致的响应。我们通过微调Qwen-2.5-3B-Instruct实现RLPA，得到Qwen-RLPA，其在个性化对话中实现了最先进的性能。实证评估显示，Qwen-RLPA在提示和离线微调基线中始终表现更优，甚至超过了高级商业模型如Claude-3.5和GPT-4o。进一步分析突显Qwen-RLPA在协调冲突用户偏好、维持长期个性化以及比最近的推理聚焦LLMs更高效推理方面的鲁棒性。这些结果强调了动态资料推断作为构建个性化对话系统更有效范式潜力。
英文摘要：
Personalized alignment is essential for enabling large language models (LLMs) to engage effectively in user-centric dialogue. While recent prompt-based and offline optimization methods offer preliminary solutions, they fall short in cold-start scenarios and long-term personalization due to their inherently static and shallow designs. In this work, we introduce the Reinforcement Learning for Personalized Alignment (RLPA) framework, in which an LLM interacts with a simulated user model to iteratively infer and refine user profiles through dialogue. The training process is guided by a dual-level reward structure: the Profile Reward encourages accurate construction of user representations, while the Response Reward incentivizes generation of responses consistent with the inferred profile. We instantiate RLPA by fine-tuning Qwen-2.5-3B-Instruct, resulting in Qwen-RLPA, which achieves state-of-the-art performance in personalized dialogue. Empirical evaluations demonstrate that Qwen-RLPA consistently outperforms prompting and offline fine-tuning baselines, and even surpasses advanced commercial models such as Claude-3.5 and GPT-4o. Further analysis highlights Qwen-RLPA's robustness in reconciling conflicting user preferences, sustaining long-term personalization and delivering more efficient inference compared to recent reasoning-focused LLMs. These results emphasize the potential of dynamic profile inference as a more effective paradigm for building personalized dialogue systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
