弥合缺失模态的差距：改进纯文本校准的视觉语言模型
Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models
AI总结：
本文提出Latent Imagination Module，通过预测文本输入的潜在嵌入来提升纯文本环境下视觉语言模型的准确性和校准性能。
摘要：
视觉语言模型（VLMs）通常在纯文本输入上部署，尽管它们是在图像上训练的。我们发现移除视觉模态会导致准确性大幅下降和严重的校准问题，且模型在纯文本提示下不再像其原始语言基座模型那样表现。这种失败不仅仅是由于缺失的语义信息。即使文本描述保留了关键内容，置信度也变得不可靠，而通过生成图像添加视觉信号部分恢复了准确性和校准。我们提出了Latent Imagination Module（LIM），一个轻量级的交叉注意力模块，它从文本输入预测想象的潜在嵌入，并将它们馈入冻结的VLM基座中，而无需像素级图像合成。在纯文本基准、未见任务和缺失图像场景中，LIM提高了准确性和减少了校准误差。这些结果表明，潜在模态的补全是可靠VLM推理在缺失模态情况下的实用方法。
英文摘要：
Vision-language models (VLMs) are often deployed on text-only inputs, although they are trained with images. We find that removing the vision modality causes large drops in accuracy and severe miscalibration, and the model does not behave like its original language backbone under text-only prompting. This failure is not explained only by missing semantic information. Even when text descriptions preserve key content, confidence becomes unreliable, while adding a visual signal through generated images partially restores accuracy and calibration. We propose the Latent Imagination Module (LIM), a lightweight cross-attention module that predicts imagined latent embeddings from textual input and feeds them into a frozen VLM backbone without pixel-level image synthesis. Across text-only benchmarks, unseen tasks, and missing-image scenarios, LIM improves accuracy and reduces calibration error. These results suggest that latent modality completion is a practical approach for reliable VLM inference under missing-modality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
