arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.26403cs.AI
收藏

从静态上下文到校准的交互式强化学习:利用对齐模拟器缓解多轮对话中的分布偏移

From Static Context to Calibrated Interactive RL: Mitigating Distribution Shift in Multi-turn Dialogue with Aligned Simulator

Fudan University(复旦大学)

Xiaohua Wang, Jiakang Yuan, Zisu Huang, Muzhao Tian, Changze Lv, Kaitao Song, Tao Chen, Xiaoqing Zheng

AI总结:

本文提出校准的交互式强化学习框架,通过将交互式强化学习与模拟器对齐相结合,缓解多轮对话中因策略和模拟器导致的分布偏移,提升对话质量。

摘要:

研究界的一个长期目标是开发高度交互的基于LLM的对话代理。最近的研究侧重于基于固定离线日志(静态上下文强化学习)或基于提示的模拟器(交互式强化学习)来优化策略。在这项工作中,我们从理论上证明,这两种范式都受到上下文分布偏移的根本限制——即训练期间观察到的对话历史与真实对话中遇到的对话历史之间的不匹配。这种偏移在每轮对话中呈二次方累积,严重降低对话质量。具体来说,我们将这种偏移归因于两个不同的来源:(i)策略引起的偏移,源于在静态历史而非自生成轨迹上进行训练;(ii)模拟器引起的偏移,源于模拟行为与真实人类行为之间的差异。为了解决这些挑战,我们提出了校准的交互式强化学习,这是一个统一的框架,将交互式强化学习与模拟器对齐相结合。通过将模拟器与人类交互模式对齐,我们的方法减少了模拟到真实的差距,并减轻了累积的分布偏移。在多个对话任务上的实验证实了我们的理论分析:(i)交互式强化学习通过缓解策略分布偏移,显著优于静态上下文基线;(ii)使用我们的对齐方法校准模拟器进一步弥合了模拟到真实的差距,产生了最先进的下游性能。

英文摘要:

A long-standing goal of the research community is to develop highly interactive LLM-based dialogue agents. Recent research focuses on optimizing policies based on fixed offline logs (Static Context RL) or using a prompt-based simulator (Interactive RL). In this work, we theoretically show that both paradigms are fundamentally limited by context distribution shift--a mismatch between dialogue histories observed during training and those encountered in real conversations. This shift compounds quadratically over turns and severely degrades dialogue quality. Specifically, we attribute this shift to two distinct sources: (i) policy-induced shift, arising from training on static histories rather than self-generated trajectories; and (ii) simulator-induced shift, stemming from discrepancies between simulated and real human behaviors. To address these challenges, we propose Calibrated Interactive RL, a unified framework that couples interactive RL with simulator alignment. By aligning the simulator with human interaction patterns, our approach reduces the sim-to-real gap and mitigates compounding distribution shifts. Experiments across multiple dialogue tasks confirm our theoretical analysis: (i) Interactive RL significantly outperforms the Static Context baseline by mitigating policy distribution shift; and (ii) calibrating simulators with our alignment method further bridges the sim-to-real gap, yielding state-of-the-art downstream performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑