arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18656cs.AI
收藏

平衡思考:改进视觉语言模型中的推理链训练

Balanced Thinking: Improving Chain of Thought Training in Vision Language Models

IBM Research(IBM研究院)

Shaked Perek, Ben Wiesel, Avihu Dekel, Nimrod Shabtay, Eli Schwartz

AI总结:

本文提出SCALe方法,通过动态权重平衡推理和答案段,提升视觉语言模型的推理效率和准确性,且训练时间仅为传统方法的七分之一。

摘要:

视觉语言模型(VLMs)的多模态推理通常依赖于监督微调(SFT)和强化学习(RL)的两阶段过程。在标准SFT中,所有token对损失贡献相等,尽管推理数据本质上是token不平衡的。长trace会掩盖短但任务关键的<answer>段,导致推理冗长且答案不准确。我们提出SCALe(Scheduled Curriculum Adaptive Loss),通过动态、长度无关的权重明确分离推理和答案段的监督。不同于传统SFT过度强调trace段,SCALe-SFT通过余弦调度策略逐步将重点从trace转移到<answer>,鼓励简洁且有根据的推理。我们评估了SCALe在多样化的基准和架构上的表现。结果表明,SCALe在准确性上优于传统SFT,并在训练时间仅为传统方法的约七分之一的情况下,与完整两阶段SFT + GRPO流程表现相当。当与GRPO结合时,SCALe实现了最佳整体性能,突显了其作为独立方法和强化精炼强大基础的价值。

英文摘要:

Multimodal reasoning in vision-language models (VLMs) typically relies on a two-stage process: supervised fine-tuning (SFT) and reinforcement learning (RL). In standard SFT, all tokens contribute equally to the loss, even though reasoning data are inherently token-imbalanced. Long <think> traces overshadow short but task-critical <answer> segments, leading to verbose reasoning and inaccurate answers. We propose SCALe (Scheduled Curriculum Adaptive Loss), which explicitly separates supervision over reasoning and answer segments using dynamic, length-independent weighting. Unlike vanilla SFT, which overweights the <think> segment, SCALe-SFT gradually shifts the focus from <think> to <answer> throughout training via a cosine scheduling policy, encouraging concise and well-grounded reasoning. We evaluate SCALe across diverse benchmarks and architectures. Results show that SCALe consistently improves accuracy over vanilla SFT and matches the performance of the full two-phase SFT + GRPO pipeline while requiring only about one-seventh of the training time, making it a lightweight yet effective alternative. When combined with GRPO, SCALe achieves the best overall performance, highlighting its value both as a standalone method and as a strong foundation for reinforcement refinement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑