GTR-Turbo：合并的检查点实际上是为代理VLM训练提供免费的教师
GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training
AI总结：
GTR-Turbo通过合并检查点作为免费教师，提升了多模态代理训练的效率和效果。
摘要：
多轮强化学习（RL）对于基于视觉语言模型（VLMs）的多模态代理而言，受到稀疏奖励和长周期信用分配的阻碍。最近的方法通过查询提供步骤级反馈的教师来密集化奖励，例如引导思维强化（GTR）和策略性蒸馏，但依赖于成本高昂、通常具有特权的模型作为教师，限制了实用性和可重复性。我们介绍了GTR-Turbo，这是对GTR的高效升级，其性能与之匹配，但无需在或查询昂贵的教师模型上进行训练。具体而言，GTR-Turbo合并了在持续RL训练过程中生成的检查点的权重，然后使用由此产生的合并模型作为“免费”的教师，通过监督微调或软logit蒸馏来指导后续的RL。这种设计消除了对特权VLMs（例如GPT或Gemini）的依赖，缓解了先前工作中观察到的“熵崩溃”现象，并保持了稳定的训练。在多样化的视觉代理任务中，GTR-Turbo在基线模型的准确性上提高了10-30%，同时将训练时间减少了50%，计算成本降低了60%。
英文摘要：
Multi-turn reinforcement learning (RL) for multi-modal agents built upon vision-language models (VLMs) is hampered by sparse rewards and long-horizon credit assignment. Recent methods densify the reward by querying a teacher that provides step-level feedback, e.g., Guided Thought Reinforcement (GTR) and On-Policy Distillation, but rely on costly, often privileged models as the teacher, limiting practicality and reproducibility. We introduce GTR-Turbo, a highly efficient upgrade to GTR that matches its performance without training on or querying an expensive teacher model. Specifically, GTR-Turbo merges the weights of checkpoints produced during ongoing RL training and then uses the resulting merged model as a "free" teacher to guide subsequent RL via supervised fine-tuning or soft logit distillation. This design removes dependence on privileged VLMs (e.g., GPT or Gemini), mitigates the "entropy collapse" observed in prior work, and maintains stable training. Across diverse visual agentic tasks, GTR-Turbo improves the accuracy of the baseline model by 10-30% while reducing wall-clock training time by 50% and compute cost by 60% relative to GTR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
