LaViDa：用于多模态理解的大型扩散语言模型
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
AI总结：
研究针对现有视觉语言模型在快速推理和可控生成方面的不足，提出基于离散扩散模型构建 LaViDa 系列视觉语言模型，采用多种新技术，在多模态基准测试中性能出色，成为自回归视觉语言模型的有力替代。
摘要：
现代视觉语言模型（VLM）可解决各种需要视觉推理的任务。在现实场景中，VLM 理想特性包括快速推理和可控生成。现有自回归 VLM 在这些方面存在困难。离散扩散模型（DM）提供了有前景的替代方案，在多模态任务中的潜力未被充分探索。我们引入基于 DM 的 LaViDa 系列 VLM，通过为 DM 配备视觉编码器并联合微调以遵循多模态指令。LaViDa 采用了如互补掩码、前缀 KV 缓存和时间步长移位等新技术。实验表明，LaViDa 在多模态基准测试中性能优于或与自回归 VLM 竞争，具有速度 - 质量权衡灵活、可控性和双向推理等优势。
英文摘要：
Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models will be released in the camera-ready version.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
