利用潜在内在反馈训练用户模拟器
Training User Simulators with Latent Intrinsic Feedback
AI总结：
本研究提出带内在反馈的用户模拟器，利用真实用户与助手的交互日志训练，其生成的评估与人类判断相关性更好，可为多轮RL训练提供有效奖励信号且计算量减半。
摘要：
用户模拟器作为训练大语言模型（LLM）助手的环境正受到关注，这类助手能更好应对与真实用户多轮对话中的细微差别。现有用户模拟研究侧重生成逼真的用户话语，却未挖掘其作为助手响应直接反馈源的潜力。这类评估信号可能未在语言中明确传达，而是存在于用户的潜在推理中。本研究引入带内在反馈的用户模拟器，即配备潜在评估状态的模型：内部思考、反应及每轮对话中以第一人称用户视角对助手的标量评分。我们利用真实用户与助手的交互日志训练基于LLM的模拟器，结果显示，与提示式模拟器基线相比，内在反馈提升了模拟逼真度与批评行为，还提供了现有微调模拟器缺失的轮级评估。此外，我们训练的模拟器生成的评估与人类判断的相关性优于同等规模提示式模拟器。最后，实验表明，带内在反馈的模拟器为多轮强化学习（RL）训练提供了有效奖励信号，其效果与依赖外部评判LLM获取奖励的设置相当，同时将RL计算量减少了一半以上。
英文摘要：
User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such evaluation signals may not be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack. Further, evaluations generated by our trained simulators correlate better with human judgments than those of prompted simulators of the same size. Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
