RLVER: 通过可验证情绪奖励的强化学习实现共情代理
RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
AI总结：
RLVER通过可验证情绪奖励提升LLM的共情能力，实验显示其在对话任务中显著提升表现，尤其在情感理解和洞察力方面成效显著。
摘要：
大型语言模型（LLMs）在逻辑和算法推理方面表现出色，但其情感智能（EQ）仍远低于其认知能力。尽管强化学习从可验证奖励（RLVR）在其他领域有所进展，但其在对话中的应用，尤其是用于情感智能，仍处于探索阶段。在本工作中，我们引入了RLVER，这是首个端到端的强化学习框架，利用模拟用户生成的可验证情绪奖励来培养LLMs的高阶共情能力。在该框架中，自洽的富有情感的模拟用户参与对话流程，并在对话中生成确定性的情绪评分，作为奖励信号指导LLM的学习。使用PPO微调公开可用的Qwen2.5-7B-Instruct模型，使其Sentient-Benchmark分数从13.3提升至79.2，同时在数学和编程能力上保持不变。广泛实验表明：（i）RLVER一致提升了多种对话能力；（ii）思考和非思考模型表现出不同的趋势——思考模型在共情和洞察力上表现突出，而非思考模型则更擅长行动；（iii）GRPO通常能带来稳定的提升，而PPO可以将某些能力推至更高水平；（iv）更具挑战性的环境并不总是更好——适度的环境往往能产生更强的结果。我们的结果表明，RLVER是实现情感智能和广泛能力的语言代理的可行途径。
英文摘要：
Large language models (LLMs) excel at logical and algorithmic reasoning, yet their emotional intelligence (EQ) still lags far behind their cognitive prowess. While reinforcement learning from verifiable rewards (RLVR) has advanced in other domains, its application to dialogue-especially for emotional intelligence-remains underexplored. In this work, we introduce RLVER, the first end-to-end reinforcement learning framework that leverages verifiable emotion rewards from simulated users to cultivate higher-order empathetic abilities in LLMs. Within this framework, self-consistent affective simulated users engage in dialogue rollouts and produce deterministic emotion scores during conversations, serving as reward signals to guide the LLM's learning. Fine-tuning publicly available Qwen2.5-7B-Instruct model with PPO boosts its Sentient-Benchmark score from 13.3 to 79.2 while largely preserving mathematical and coding competence. Extensive experiments reveal that: (i) RLVER consistently improves multiple dialogue capabilities; (ii) Thinking and non-thinking models show distinct trends--thinking models excel in empathy and insight, while non-thinking models favor action; (iii) GRPO often yields stable gains, while PPO can push certain capabilities to a higher ceiling; (iv) More challenging environments are not always better-moderate ones can yield stronger outcomes. Our results show that RLVER is a practical route toward emotionally intelligent and broadly capable language agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
