通过言语反馈强化人类行为模拟
Reinforcing Human Behavior Simulation via Verbal Feedback
AI总结：
本文提出DITTO模型，通过将言语反馈作为强化学习中的首要信号来提升LLM模拟人类行为的能力，并引入SOUL基准测试平台，展示了在多个任务中显著提升性能的成果。
摘要：
人类通过言语反馈（例如父母说“那很粗鲁”或朋友解释“这是为什么那会伤害你”）学习社会规范和行为。然而，对于LLM而言，学习反馈主要集中在代码和数学等领域，这些领域中的RL奖励可以直接验证并压缩为标量值。随着LLM越来越多地用于模拟人类行为，例如代表用户、患者、学生和其他角色，有必要使它们更加人性化，这需要接受一种根本不同的信号：主观的、多方面的言语反馈。我们提出了DITTO，一个通过将言语反馈作为强化学习中的首要信号进行训练的模型。每次回放后，DITTO会接收言语反馈并生成反馈条件的改进回放；两个输出通过GRPO联合优化，将言语指导蒸馏到基础策略中，而无需在测试时使用反馈。我们还引入了SOUL（Simulation gym Of hUman-Like behavior），一个涵盖10个任务、六个类别的统一基准和训练数据集：理论思维、角色扮演、社交技能、学习模拟、用户模拟和角色模拟。DITTO在基础模型上平均提升了36%，并在SOUL基准测试中的6个任务上超过了GPT-5.4，证明了通过言语反馈的强化学习是训练LLM模拟人类行为的有前途的方向。
英文摘要：
Humans learn social norms and behaviors from verbal feedback (e.g., a parent saying "that was rude" or a friend explaining "here's why that hurt"). Yet, learning from feedback for LLMs has largely focused on domains like code and math, where RL rewards are directly verifiable and condensed into scalar values. As LLMs are increasingly used to simulate human behavior, e.g., standing in for users, patients, students, and other personas, there is a pressing need to make them more human-like, which requires embracing a fundamentally different kind of signal: feedback that is verbal, subjective, and multi-faceted. We present DITTO, a model trained by treating verbal feedback as a first-class signal in reinforcement learning. After each rollout, DITTO receives verbal feedback and generates a feedback-conditioned improved rollout; both outputs are jointly optimized with GRPO, distilling verbal guidance into the base policy without requiring feedback at test time. We also introduce SOUL (Simulation gym Of hUman-Like behavior), a unified benchmark and training data suite spanning 10 tasks across six categories: Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation. DITTO achieves an average 36% improvement over the base model and exceeds GPT-5.4 on 6 of 10 SOUL benchmarks, demonstrating that RL with verbal feedback is a promising direction for training LLMs to simulate human behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
