标记思维：通过强化学习释放个性化推理能力
TagPR: Tag-Guided Process Supervision for Personalization Reasoning in Large Language Models
AI总结：
针对LLM个性化推理不足，提出TagPR框架，通过语义标注推理链、SFT与多阶段RL及PRMU复合奖励实现用户逻辑对齐，并在LaMP等数据集上取得SOTA。
摘要：
近期进展赋予了大型语言模型（LLM）令人瞩目的通用推理能力，然而它们在个性化推理方面仍常显吃力——个性化推理是分析用户历史、推断独特偏好并生成定制化回复的关键能力。为解决这一局限，我们提出 TagPR，这是一种新颖的训练框架，通过“标记思维”方法显著增强 LLM 的内在个性化推理能力。我们的方法首先构建数据驱动流程，自动生成推理链并对其进行语义标注，从而创建促进可解释推理的结构化数据集。随后，我们提出协同训练策略：先在该带标签数据上进行监督微调（SFT），建立基础推理模式，再开展多阶段强化学习（RL）。该 RL 阶段由独特的复合奖励信号引导，该信号整合基于标签的约束以及新颖的带用户嵌入的个性化奖励模型（PRMU），以实现与用户特定逻辑的细粒度对齐。在公开 LaMP 基准和自建数据集上的大量实验表明，我们的方法取得了最先进结果，在所有任务上较基础模型平均提升 32.65%。本研究验证了结构化、可解释的推理是释放 LLM 真正个性化能力的高效途径。
英文摘要：
Recent advancements have endowed Large Language Models with impressive general reasoning capabilities. However, these reasoning models often perform worse than non-reasoning models on personalization tasks. While some methods use outcome-based RL to improve personalization reasoning, they fail to supervise the reasoning process. As a result, models may reach correct answers through flawed reasoning chains, limiting further improvement. To address this, we propose TagPR, a novel framework that adds semantic tags to the reasoning process for step-by-step guidance. TagPR first automatically generates a structured, tagged dataset for Supervised Fine-Tuning. It then employs a multi-stage RL process guided by a composite reward signal, which integrates tag-based process supervision with a novel Personalization Reward Model with User Embeddings to achieve fine-grained alignment with user-specific logic. Extensive experiments on public LaMP, LongLaMP, PGraphRAG, and a self-constructed dataset demonstrate that our approach achieves state-of-the-art results, delivering an average improvement of 32.65% over the base model across all LaMP benchmark tasks. Our work demonstrates that tag-guided process supervision is an effective approach for personalization reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
