VaME：探索多模态嵌入的变分潜在推理

中文摘要 AI 辅助

通用多模态检索需要紧凑的嵌入，以在跨不同模态中保留与任务相关的语义信息。先前的工作已将潜在推理融入多模态嵌入学习，以在嵌入提取前细化这些信息。然而，大多数现有方法仍局限于确定性潜在路径，未探索替代轨迹以发现更好的嵌入。因此，我们提出VaME（变分多模态嵌入），一个将潜在推理建模为轨迹上可学习分布的框架。具体而言，我们首先引入变分潜在推理（VLR），通过轻量级解码器的答案重建引导，在潜在空间中进行自回归探索。同时，我们用潜在融合嵌入增强原始嵌入-标记读出，以促进后续强化学习中的探索。最后，我们通过强化学习优化随机变分轨迹上的潜在推理，使用语义解码奖励（SDR）来偏好具有可解释解码结果的语义有意义轨迹。在涵盖图像、视频和视觉文档检索的78任务MMEB-V2基准上，VaME优于大多数显式CoT模型和所有潜在推理基线。VaME在MRMR等推理密集型基准上也表现出稳健性能，在强化学习后获得显著提升。重要的是，VaME在实现这些提升的同时，相比确定性潜在自回归基线至少获得4.25倍的推理加速。代码将公开提供。

英文摘要

Universal multimodal retrieval requires compact embeddings that preserve task-relevant semantic information across diverse modalities. Prior works have incorporated latent reasoning into multimodal embedding learning to refine this information before embedding extraction. However, most existing approaches remain confined to deterministic latent paths, without exploring alternative trajectories to discover better embeddings. Thus, we propose VaME (Variational Multimodal Embeddings), a framework that models latent reasoning as a learnable distribution over trajectories. Specifically, we first introduce Variational Latent Reasoning (VLR) to enable autoregressive exploration in latent space, guided by answer reconstruction through a lightweight decoder. Meanwhile, we augment the original embedding-token readout with a latent-fused embedding to facilitate exploration during subsequent reinforcement learning. Finally, we optimize latent reasoning over stochastic variational trajectories through reinforcement learning, using Semantic Decoding Reward (SDR) to favor semantically meaningful trajectories with interpretable decoded outcomes. On the 78-task MMEB-V2 benchmark, spanning image, video, and visual-document retrieval, VaME outperforms most explicit CoT-based models and all latent-reasoning baselines. VaME also demonstrates robust performance on reasoning-intensive benchmarks such as MRMR, with substantial gains after reinforcement learning. Importantly, VaME achieves these gains with at least a 4.25x inference speedup over the deterministic latent autoregressive baselines. The code will be made publicly available.

发表机构

• University of Science and Technology of China（中国科学技术大学）
• Tsinghua University（清华大学）
• Kuaishou（快手）
• Zhejiang University（浙江大学）

机构由 AI 辅助整理，请以论文原文为准。