arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.18641cs.CV
收藏

利用沉默中的潜在视觉推理

Leveraging Latent Visual Reasoning in Silence

North Carolina State University(北卡罗来纳州立大学);UC, San Diego(加州大学圣地亚哥分校);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Johns Hopkins University(约翰霍普金斯大学);Duke University(杜克大学);Boston University(波士顿大学);The Ohio State University(俄亥俄州立大学)

Dongyao Zhu, Zhen Wang, Xi Xiao, Han Jiang, Saeed Vahidian, Wei-Lun Chao, Tanya Berger-Wolf, Yu Su, Raju Vatsavai, Jianyang Gu

AI总结:

本文探讨了在推理过程中是否需要持续的潜在令牌,发现即使移除这些令牌或用随机噪声替代,性能影响较小,提出了一种基于注意力的奖励机制以促进潜在令牌与后续文本令牌的交互,从而提升视觉感知和视觉推理任务的性能。

摘要:

潜在视觉推理通过在文本生成前插入连续潜在令牌,更直接地参与多模态推理。然而,这些潜在令牌在推理中的必要性仍存疑。我们发现,在空间推理基准上,用随机噪声替代或完全移除潜在令牌对性能影响很小。强化学习进一步在训练后减少了潜在生成行为。这些观察引发了一个核心问题:潜在视觉推理是否仍然有意义?我们认为其价值应由潜在令牌如何引导学习来衡量,而非是否在推理时保留。我们的分析表明,潜在推理在不同问题类型中效果不均,但任务级路由应用潜在生成是脆弱的。受这些发现启发,我们提出了一种基于注意力的奖励,鼓励生成的潜在令牌在强化学习中与后续文本令牌交互。该奖励在潜在模式激活时促进潜在利用,同时保持使用纯文本推理的灵活性。实验表明,我们的方法在感知和视觉推理基准上提升了性能,即使在训练后潜在令牌很少生成。我们的结果表明,在推理时没有显式表达的情况下,潜在视觉推理可以塑造更好的视觉基础和更准确的文本推理。我们的代码和训练模型可在GitHub和Hugging Face上公开获取。

英文摘要:

Latent visual reasoning involves visual evidence more directly in multimodal reasoning by inserting continuous latent tokens before textual generation. However, the necessity of these latent tokens at inference remains ambiguous. We show that replacing latent tokens with random noise or removing them completely causes little performance degradation across spatial reasoning benchmarks. Reinforcement learning further diminishes the latent generation behavior after post-training. These observations raise a central question: \textit{Is latent visual reasoning still meaningful}? We argue that its value should be measured by how effectively latent tokens guide learning, rather than whether they persist as an inference-time format. Our analysis shows that latent reasoning is unevenly favorable across question types, yet hard task-level routing for applying latent generation is brittle. Motivated by these findings, we propose an attention-based reward that encourages generated latent tokens to interact with later text tokens during RL. This reward promotes latent utilization when the latent mode is activated while preserving the flexibility to use pure-text reasoning. Experiments show that our method improves performance across perception and visual reasoning benchmarks, even when latent tokens are rarely generated after post-training. Our results highlight that, without explicit expression at inference, latent visual reasoning can shape better visual grounding and more accurate textual reasoning \textbf{in silence}. Code \& model: \href{https://github.com/ddydyd32/silent-lvr/tree/master}{\faGithub}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑