VLIS：单模态语言模型引导多模态语言生成
VLIS: Unimodal Language Models Guide Multimodal Language Generation
AI总结：
针对现有视觉-语言模型复杂语言理解能力不足的问题，提出无需额外训练的VLIS框架，以视觉-语言模型提取的图-文逐点互信息为重要性采样权重调整纯文本模型词元似然，在多类任务上取得提升，为多模态语言生成提供新方向。
摘要：
多模态语言生成利用语言与视觉的协同效应，是一个快速发展的领域。然而，现有视觉-语言模型在需要复杂语言理解的任务中面临挑战。为解决这一问题，我们提出了Visual-Language models as Importance Sampling weights（VLIS，视觉-语言模型作为重要性采样权重）这一新颖框架，它无需额外训练，就能结合视觉-语言模型的视觉条件处理能力与单模态纯文本语言模型的语言理解能力。该框架从视觉-语言模型中提取每张图像与文本的逐点互信息，并将该值用作重要性采样权重，以调整纯文本模型输出的词元似然。VLIS在多种任务上提升了视觉-语言模型的表现，包括常识理解任务（WHOOPS、OK-VQA和ScienceQA）以及复杂文本生成任务（Concadia、图像段落描述和ROCStories）。我们的结果表明，VLIS为多模态语言生成提供了一个颇具前景的新方向。
英文摘要：
Multimodal language generation, which leverages the synergy of language and vision, is a rapidly expanding field. However, existing vision-language models face challenges in tasks that require complex linguistic understanding. To address this issue, we introduce Visual-Language models as Importance Sampling weights (VLIS), a novel framework that combines the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without further training. It extracts pointwise mutual information of each image and text from a visual-language model and uses the value as an importance sampling weight to adjust the token likelihood from a text-only model. VLIS improves vision-language models on diverse tasks, including commonsense understanding (WHOOPS, OK-VQA, and ScienceQA) and complex text generation (Concadia, Image Paragraph Captioning, and ROCStories). Our results suggest that VLIS represents a promising new direction for multimodal language generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
