arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04783cs.AIcs.CL
收藏

打破情境惯性：基于单轮锚点的强化学习用于稳定多轮交互

Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction

Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）

Xingwu Chen, Zhanqiu Zhang, Yiwen Guo, Difan Zou

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出RLSTA方法，通过单轮锚点提供奖励信号，帮助模型打破情境惯性，提升多轮交互稳定性，实验显示其在不同领域表现优异。

摘要：

尽管LLM在单轮提供完整信息时表现出色，但在多轮交互中却存在显著脆弱性。当信息逐步揭示或需更新时，模型常无法整合新约束，导致性能下降。我们将其根本原因称为情境惯性：模型固守先前推理轨迹。即使用户在后续轮次提供修正或新数据，模型也忽略这些信息，倾向于维持一致性。为此，我们引入RLSTA，一种通用训练方法，旨在稳定多轮交互。RLSTA利用模型的单轮能力作为稳定内部锚点，提供奖励信号。通过将多轮响应与这些锚点对齐，RLSTA使模型能够打破情境惯性并根据最新信息自我校准。实验表明，RLSTA显著优于标准微调和回避方法。值得注意的是，我们的方法在跨领域泛化（如数学到代码）中表现强劲，并且即使没有外部验证器也有效，展示了其在通用领域应用的潜力。代码可在https://github.com/Tencent/RLSTA获取。

英文摘要：

While LLMs demonstrate strong reasoning capabilities when provided with full information in a single turn, they exhibit substantial vulnerability in multi-turn interactions. Specifically, when information is revealed incrementally or requires updates, models frequently fail to integrate new constraints, leading to a collapse in performance compared to their single-turn baselines. We term the root cause as \emph{Contextual Inertia}: a phenomenon where models rigidly adhere to previous reasoning traces. Even when users explicitly provide corrections or new data in later turns, the model ignores them, preferring to maintain consistency with its previous (incorrect) reasoning path. To address this, we introduce \textbf{R}einforcement \textbf{L}earning with \textbf{S}ingle-\textbf{T}urn \textbf{A}nchors (\textbf{RLSTA}), a generalizable training approach designed to stabilize multi-turn interaction across diverse scenarios and domains. RLSTA leverages the model's superior single-turn capabilities as stable internal anchors to provide reward signals. By aligning multi-turn responses with these anchors, RLSTA empowers models to break contextual inertia and self-calibrate their reasoning based on the latest information. Experiments show that RLSTA significantly outperforms standard fine-tuning and abstention-based methods. Notably, our method exhibits strong cross-domain generalization (e.g., math to code) and proves effective even without external verifiers, highlighting its potential for general-domain applications. Code is available at https://github.com/Tencent/RLSTA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑