量化用户模拟器在构建协作大语言模型助手中的效用
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
AI总结:
本文通过实验评估用户模拟器的质量,发现基于细调模拟器训练的助手在真实人类交互中表现更优,且模拟器规模扩大对细调模拟器有效,但对角色扮演模拟器无帮助。
摘要:
用户模拟器被越来越多地用于构建交互式AI助手,但如何衡量这些模拟器的质量仍是一个开放问题。本文展示了如何通过下游效用量化模拟器质量:训练LLM助手时,其在真实人类交互中的表现。在控制实验中,我们训练LLM助手对抗不同模拟器,从提示LLM扮演用户到细调于WildChat的人类对话。评估通过283名参与者参与的用户研究和WildBench基准测试进行。使用角色扮演LLM训练的助手在用户研究中与初始助手无显著差异(51%胜率),而使用细调模拟器训练的助手表现显著提升(58%优于初始,57%优于角色扮演训练的助手)。进一步分析显示,使角色扮演LLM更真实的策略(如身份条件)虽能提升助手表现,但无法缩小与细调模拟器的差距;扩大模拟器模型规模对细调模拟器有益,但对角色扮演模拟器无帮助;而训练于角色扮演模拟器的助手在与其他模拟器配对时无法泛化,而训练于细调模拟器的助手则能。这些结果表明,应基于真实人类行为来构建用户模拟器,并通过其对真实用户的影响来衡量其质量。
英文摘要:
User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs in the wild when interacting with real humans. In a controlled experiment where only the user simulator varies, we train LLM assistants via reinforcement learning against a spectrum of simulators, from an LLM prompted to role-play a user to one fine-tuned on human utterances from WildChat. As evaluation, we measure pairwise win rates in a user study with 283 participants and on WildBench, a benchmark derived from real human--AI conversations. Training against the role-playing LLM yields an assistant statistically indistinguishable from the initial assistant in our user study (51% win rate), whereas training against the fine-tuned simulator yields significant gains (58% over the initial and 57% over the one trained against role-playing). Closer inspection reveals three further patterns: methods for making role-playing LLMs more realistic (e.g., persona conditioning) improve trained assistants but do not close the gap to the fine-tuned simulator; scaling the simulator's model size benefits the fine-tuned simulator but yields no gain for role-playing ones; and assistants trained against role-playing simulators fail to generalize when paired with other simulators at test time, while the one trained against fine-tuned simulator does. Together, these results argue for grounding user simulators in real human behavior and measuring their quality by their downstream effect on real users.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。