arXiv 2512.025362025-12-02cs.CV

WeMMU: 通过噪声查询标记增强视觉-语言模型与扩散模型的桥梁

WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（脑启发智能感知与认知联合实验室，中国科学技术大学）；ZheJiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）

Jian Yang, Dacheng Yin, Xiaoxuan He, Yong Li, Fengyun Rao, Jing Lyu, Wei Zhai, Yang Cao, Zheng-Jun Zha

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

WeMMU通过噪声查询标记和VAE分支，提升视觉-语言模型与扩散模型的连接效率，缓解泛化崩溃问题，实现稳定持续学习。

摘要：

近年来，多模态大语言模型（MLLMs）的进步突显了高效地将预训练视觉-语言模型（VLMs）与扩散模型连接起来的挑战。虽然使用固定数量可学习查询标记的方法在计算效率上有所优势，但它们在任务泛化上会崩溃，无法适应与预训练任务相距较远的新任务。为了解决这一问题，我们提出了噪声查询标记，通过端到端优化在VLM和扩散模型之间学习分布式表示空间，增强持续学习。此外，我们引入了一个带有线性投影的VAE分支以恢复细粒度图像细节。实验结果证实我们的方法减轻了泛化崩溃，并在多样化的任务上实现了稳定的持续学习。

英文摘要：

Recent progress in multimodal large language models (MLLMs) has highlighted the challenge of efficiently bridging pre-trained Vision-Language Models (VLMs) with Diffusion Models. While methods using a fixed number of learnable query tokens offer computational efficiency, they suffer from task generalization collapse, failing to adapt to new tasks that are distant from their pre-training tasks. To overcome this, we propose Noisy Query Tokens, which learn a distributed representation space between the VLM and Diffusion Model via end-to-end optimization, enhancing continual learning. Additionally, we introduce a VAE branch with linear projection to recover fine-grained image details. Experimental results confirm our approach mitigates generalization collapse and enables stable continual learning across diverse tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。