PEER：统一的过程-结果强化学习用于结构化共情推理
PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoning
AI总结：
本文提出PEER，通过结构化共情推理框架提升情感支持对话的 empathy、策略一致性及人性表现，采用GRPO与UnifiReward模型，结合数据增强减少重复。
摘要：
情感支持对话需要更 than流畅回应。支持者需理解寻求者的处境与情绪，采用适当策略并以自然人性化方式回应。尽管大语言模型取得进展，当前系统常缺乏结构化、心理学导向的推理。此外，通过强化学习提升这些系统具有挑战，因奖励信号不可靠。此外，强化微调可能放大重复回应模式。我们提出结构化共情推理，将其分解为三个步骤：对话历史分析、多模态情绪状态推断和策略选择，然后生成最终回复。为实现此目标，我们引入SER，一个细粒度数据集，包含步骤级正确性标签和成对回应偏好。我们随后提出PEER，使用GRPO与UnifiReward，一个统一的过程-结果奖励模型，用于评估多轮交互中推理步骤和最终回应。为减少重复，我们通过基于个性的重写增强数据，并降低冗余输出权重。全面实验表明，PEER在不牺牲多样性的情况下，提升了共情、策略一致性和人性表现。
英文摘要：
Emotional support conversations require more than fluent responses. Supporters need to understand the seeker's situation and emotions, adopt an appropriate strategy, and respond in a natural, human-like manner. Despite advances in large language models, current systems often lack structured, psychology-informed reasoning. Additionally, it is challenging to enhance these systems through reinforcement learning because of unreliable reward signals. Moreover, reinforcement fine-tuning can amplify repetitive response patterns. We propose structured empathetic reasoning, which breaks support into three steps: conversation history analysis, multimodal emotional state inference, and strategy selection, prior to generating the final reply. To implement this, we introduce SER, a fine-grained dataset with step-level correctness labels and pairwise response preferences. We then present PEER, which uses GRPO with UnifiReward, a unified process-outcome reward model for evaluating both reasoning steps and final responses in multi-turn interactions. To reduce repetition, we enhance data with personality-based rewriting and down-weight redundant outputs. Comprehensive experiments show improved empathy, strategy alignment, and human-likeness without sacrificing diversity. Code and data are available at https://github.com/Yunxiao-Wang/PEER.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
