发表机构
The Hong Kong University of Science and Technology (Guangzhou); Tencent; Shanghai Jiao Tong University（香港科技大学（广州）; 腾讯; 上海交通大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对角色扮演智能体训练中固定场景池导致的分布瓶颈，提出AdvRole对抗性上下文重写框架，通过性能差距奖励训练Rewriter生成困难场景，实现闭环课程，实验证明其在多语言基准上优于基线。
AI 中文摘要
基于大语言模型的角色扮演智能体已广泛应用于个性化辅助和社会模拟等领域。现有的强化学习方法通常在训练开始前收集的固定场景池上进行训练。这造成了分布瓶颈：随着智能体能力的提升，其表现不佳的场景也在变化，而训练分布保持不变。为此，我们提出AdvRole，一种对抗性上下文重写框架，将角色扮演强化学习转变为闭环课程。AdvRole交替训练一个学习角色扮演的Actor和一个将角色档案与对话上下文改写为针对该Actor的困难场景的Rewriter。Rewriter使用性能差距奖励进行训练，该奖励倾向于选择那些能降低当前Actor相对于原始场景得分的重写。因此，场景池随Actor的演化而动态更新，并持续针对角色-上下文空间中未被充分掌握的区域。在覆盖英文和中文的三个角色扮演基准测试以及我们发布的一个新的多语言基准测试上的实验表明，AdvRole持续优于基线方法。
英文摘要
Role-playing agents based on large language models have been widely applied in areas such as personalized assistance and social simulation. Recent RL methods typically train on a fixed scenario pool collected before learning begins. This creates a distributional bottleneck: as the agent improves, the scenarios where it performs poorly also change, while the training distribution remains static. Therefore, we propose AdvRole, an adversarial context rewriting framework that turns role-playing RL into a closed-loop curriculum. AdvRole alternates between an Actor that learns to role-play and a Rewriter that edits character profiles and dialogue contexts into actor-specific hard scenarios. The Rewriter is trained with a performance-gap reward, which favors rewrites that reduce the current Actor's score relative to the original scenario. As a result, the scenario pool evolves with the Actor and continuously targets under-mastered regions of the character-context space. Experiments on three role-playing benchmarks covering English and Chinese, as well as a new multilingual benchmark we release, show that AdvRole consistently outperforms baselines.
