arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08533cs.AI
收藏

通过智能体游戏和自适应树形GRPO优化对话模型

Dialogue Model Optimization via Agent Game and Adaptive Tree-based GRPO

Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Tencent（腾讯）

Kun Peng, Conghui Tan, Yu Liu, Guohua Tang, Zhongqian Sun, Wei Yang, Zining Zhu, Lei Jiang, Yanbing Liu, Hao Peng

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种结合智能体游戏和自适应树形GRPO的对话模型优化方法，通过动态环境构建和自适应奖励范围设计，提升对话的长期价值捕获与样本效率。

摘要：

开放式对话代理旨在通过适应用户特征来提供引人入胜、个性化的互动，但现有方法面临关键限制：过度依赖预收集的用户数据，以及强化学习（RL）中的短视偏差，忽视长期对话价值。为了解决这些问题，我们提出了一种新的长视距RL框架，整合在线个性化与自适应树形组相对策略优化（AT-GRPO）。采用双智能体游戏范式，用户智能体通过风格模仿（学习用户特定的对话特征）和主动终止（预测回合级终止概率作为即时奖励）构建动态环境，形成一个迭代循环，推动对话代理深入兴趣探索。AT-GRPO将对话轨迹重新解释为树，并引入自适应观察范围。不同于完全树扩展导致指数级开销，它限制每个节点仅聚合来自阶段感知范围内的奖励：较大的范围支持早期阶段话题探索，而较小的范围促进后期阶段对话维护。此设计将 rollout 预算从指数级降低到对话长度的多项式级，同时保持长期奖励捕获。大量实验显示我们的框架在性能、样本效率和鲁棒性方面均优于现有方法。

英文摘要：

Open-ended dialogue agents aim to deliver engaging, personalized interactions by adapting to users' traits, but existing methods face critical limitations: over-reliance on pre-collected user data, and short-horizon biases in reinforcement learning (RL) that neglect long-term dialogue value. To address these, we propose a novel long-horizon RL framework integrating online personalization with Adaptive Tree-based Group Relative Policy Optimization (AT-GRPO). Adopting a two-agent game paradigm, a user agent constructs dynamic environments via style mimicry (learning user-specific conversational traits) and active termination (predicting turn-level termination probabilities as immediate rewards), forming an iterative cycle that drives the dialogue agent to deepen interest exploration. AT-GRPO reinterprets dialogue trajectories as trees and introduces adaptive observation ranges. Unlike full tree expansion that incurs exponential overhead, it limits each node to aggregate rewards from a stage-aware range: larger ranges support early-stage topic exploration, while smaller ranges facilitate late-stage dialogue maintenance. This design reduces rollout budgets from exponential to polynomial in the dialogue length, while preserving long-term reward capture. Extensive experiments show our framework's superior performance, sample efficiency, and robustness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑