TurnSight：面向工具集成推理的回合级事后自蒸馏方法
TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning
浏览论文内容
中文总结 AI 辅助
针对工具集成推理中现有方法缺乏细粒度监督的问题，提出TurnSight框架，通过回合级事后自蒸馏生成可靠监督信号，在三个基准上验证了其有效性。
中文摘要 AI 辅助
工具集成推理（TIR）使大语言模型（LLM）能通过迭代式工具交互解决复杂任务，但现有强化学习方法常依赖轨迹级监督，限制了长时序TIR场景下的细粒度信用分配。同策略自蒸馏通过带特权上下文的教师分支提供更密集的监督信号，不过现有方法通常从真实答案或检索技能中获取此类上下文，可能无法反映智能体实际访问的状态，且令牌级监督无法捕捉工具交互的回合级结构。为解决该问题，本文提出TurnSight，一种回合级事后自蒸馏框架，直接从执行条件事后经验中获取监督信号，随后构建多个具有不同前瞻范围的事后视图，并通过跨范围方向一致性选择可靠的监督信号，最后将所选事后信号在并行轨迹中归一化，用于自适应调整强化学习优势值，同时保留其原始优化方向。在三个基准上开展的大量实验验证了TurnSight的有效性，代码可在指定网址获取。
英文摘要
Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.
