双潜记忆路由用于视觉-语言推理
Dual-Latent Memory Routing for Vision-Language Reasoning
- Nanjing University（南京大学）
- Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对多模态大模型长序列推理中视觉证据丢失问题，提出双潜记忆路由机制，通过视觉与推理记忆分离及动态路由，以少量参数提升推理性能。
AI中文摘要：
多模态大语言模型（MLLMs）近期在视觉-语言推理方面取得了显著进展，然而随着生成序列变长，其性能往往会下降。一个关键因素是，在单一不断增长的上下文环境中，它们经常丢失早期视觉证据和中间约束。受人类在解决复杂任务时分别回忆所见与所推之理的启发，我们提出DLMR，一种参数高效的机制，为MLLMs配备双潜记忆：一个压缩图像证据的视觉记忆和一个追踪中间结论与约束的推理记忆。随后，一个路由器在推理过程中动态决定使用哪个记忆以及使用多少，从而在保持视觉基础的同时维持连贯的长程推理。DLMR分三个阶段训练，从潜记忆构建到选择性路由器学习，同时保持基础MLLM冻结，仅用少量额外可训练参数即在通用和推理基准上取得显著提升。分析进一步展示了可解释的、状态依赖的路由，具有专门的记忆角色，并减少了长生成过程中的解码令牌。代码可在以下网址获取：此https URL。
英文摘要：
Multimodal large language models (MLLMs) have recently made strong progress in vision-language reasoning, yet their performance often degrades as generations grow longer. A key factor is that they frequently lose track of earlier visual evidence and intermediate constraints under a monolithic growing context. Inspired by how humans separately recall what they see and what they infer when solving complex tasks, we propose DLMR, a parameter-efficient mechanism that equips MLLMs with Dual Latent Memories: a visual memory that compresses image evidence and a reasoning memory that tracks intermediate conclusions and constraints. A Router then dynamically decides which memory and how much to reuse during inference, preserving visual grounding while maintaining coherent long-horizon reasoning. DLMR is trained in three stages, from latent memory construction to selective router learning, while keeping the base MLLM frozen, yielding substantial gains on both general and reasoning benchmarks with only a small number of additional trainable parameters. Analyses further show interpretable, state-dependent routing with specialized memory roles and reduced decoding tokens over long generations. Code is available at https://github.com/Hunter-Wrynn/DLMR.
