arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.00461cs.CV
收藏

非对称互变分学习实现多模态连续推理

Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning

Shanghai Jiao Tong University(上海交通大学);Ant Group(蚂蚁集团)

Shijie Li, Yilin Gao, Siyuan Yang, Tieyuan Chen, Chaofan Gan, Zhihao He, Zicheng Zhao, Yuyu Guo, Weiyao Lin, Hang Yu

AI总结:

针对多模态大语言模型离散推理丢失感知细节及训练-推理不匹配问题,提出非对称互变分学习框架,通过双向KL散度校准先验与后验,缓解答案泄露,提升连续潜在推理性能。

摘要:

多模态大语言模型(MLLMs)常受限于语言空间瓶颈,将复杂的视觉推理强制转化为离散标记,从而丢失感知细节。一种有前景的替代方案是连续潜在推理,其目标是发现连接多模态查询与最终答案的隐式推理路径。然而,这引入了严重的训练-推理不匹配:依赖于真实答案的训练时后验会利用答案相关的捷径。标准的变分训练迫使推理时的先验去模仿一个能够访问测试时不可用信息的后验,导致性能不佳。为解决此问题,我们提出非对称互变分学习(AMVL),一种通过双向校准目标解决此不匹配的框架。前向KL散度训练与答案无关的先验去匹配后验,而新颖的反向KL散度同时正则化后验,防止其坍缩到推理不兼容的区域,并缓解这种“答案泄露”。我们提供了理论分析,将这种泄露形式化为先验污染,并证明我们的双KL目标能够减少它。我们将AMVL实例化到一个潜在集成的MLLM中,并展示它持续优于强离散和潜在推理基线,在复杂BLINK基准上的平均得分提高了+10.83,在单个推理任务上取得了高达+32.00的提升,分析证实了潜在空间稳定性的改善。

英文摘要:

Multimodal Large Language Models (MLLMs) are often constrained by a language-space bottleneck, forcing complex visual reasoning into discrete tokens which can lose perceptual nuance. A promising alternative is continuous latent reasoning, where the goal is to discover implicit reasoning pathways that bridge the multimodal query and the final answer. However, this introduces a severe train-inference mismatch: a training-time posterior, conditioned on the ground-truth answer, can exploit answer-dependent shortcuts. Standard variational training then forces the inference-time prior to mimic a posterior that has access to information unavailable at test time, leading to poor performance. To address this, we propose Asymmetric Mutual Variational Learning (AMVL), a framework that resolves this mismatch via a bidirectional calibration objective. A forward KL divergence trains the target-agnostic prior to match the posterior, while a novel reverse KL divergence simultaneously regularizes the posterior, preventing it from collapsing into inference-incompatible regions and mitigating this ``answer leakage''. We provide theoretical analysis formalizing this leakage as prior contamination and prove that our dual-KL objective reduces it. We instantiate AMVL in a latent-integrated MLLM and show that it consistently outperforms strong discrete and latent-reasoning baselines, improving the average score on the complex BLINK benchmark by +10.83 and achieving gains of up to +32.00 on individual reasoning tasks, with analyses confirming improved latent-space stability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑