面向视觉推理的潜思维链
Latent Chain-of-Thought for Visual Reasoning
AI总结：
针对大视觉语言模型现有推理训练算法泛化差、依赖偏差奖励模型的问题，提出基于变分推断的潜思维链训练方法，结合稀疏奖励与贝叶斯推断缩放策略，在7个基准上提升了模型推理的有效性、泛化性与可解释性。
摘要：
思维链（Chain-of-thought, CoT）推理对于提升大视觉语言模型（Large Vision-Language Models, LVLMs）的可解释性与可靠性至关重要。然而，现有训练算法如SFT、PPO和GRPO在未见过的推理任务上泛化性不佳，且严重依赖存在偏差的奖励模型。为解决这一挑战，我们将LVLMs中的推理重新表述为后验推断，并提出一种基于 amortized 变分推断的可扩展训练算法。通过利用追求多样性的强化学习算法，我们提出一种新颖的稀疏奖励函数，为 token 级学习信号提供激励，催生多样化、高似然的潜CoT，克服确定性采样的局限性并避免奖励黑客。此外，我们实现了一种贝叶斯推断缩放策略，用边际似然替代成本高昂的Best-of-N和Beam Search，高效排序最优推理依据与答案。我们通过实验证明，所提方法在7个推理基准上提升了当前最优LVLM的有效性、泛化性与可解释性。
英文摘要：
Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs). However, existing training algorithms such as SFT, PPO, and GRPO may not generalize well across unseen reasoning tasks and heavily rely on a biased reward model. To address this challenge, we reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference. By leveraging diversity-seeking reinforcement learning algorithms, we introduce a novel sparse reward function for token-level learning signals that encourage diverse, high-likelihood latent CoT, overcoming deterministic sampling limitations and avoiding reward hacking. Additionally, we implement a Bayesian inference-scaling strategy that replaces costly Best-of-N and Beam Search with a marginal likelihood to efficiently rank optimal rationales and answers. We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks, in terms of effectiveness, generalization, and interpretability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
