视觉思维链：用多模态填充弥合逻辑缺口
Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings
AI总结：
本文提出 VCoT 方法，通过视觉-语言 grounding 与思维链提示递归生成多模态填充内容，以弥合序列数据中的逻辑缺口，并在 Visual Storytelling 和 WikiHow 数据集上验证其优于思维链基线的数据增强效果。
摘要：
大型语言模型的最新进展引发了思维链推理，使模型能够以类似人类的方式分解问题。尽管这一范式提升了语言模型的多步推理能力，但它受限于单模态特性，并且主要应用于问答任务。我们认为，将视觉增强融入推理至关重要，尤其是对于复杂的、富有想象力的任务。因此，我们提出了 VCoT，一种利用思维链提示与视觉-语言 grounding 来递归地弥合序列数据中逻辑缺口的新方法。我们的方法使用视觉引导来生成合成的多模态填充内容，这些填充内容添加了一致且新颖的信息，以减少下游任务中的逻辑缺口，从而使下游任务能够受益于时间推理，同时为模型的多步推理提供可解释性。我们将 VCoT 应用于 Visual Storytelling 和 WikiHow 摘要数据集，并通过人工评估证明，VCoT 提供了新颖且一致的合成数据增强，优于思维链基线，可用于提升下游性能。
英文摘要：
Recent advances in large language models elicit reasoning in a chain-of-thought that allows models to decompose problems in a human-like fashion. Though this paradigm improves multi-step reasoning ability in language models, it is limited by being unimodal and applied mainly to question-answering tasks. We claim that incorporating visual augmentation into reasoning is essential, especially for complex, imaginative tasks. Consequently, we introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to recursively bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks that can benefit from temporal reasoning, as well as provide interpretability into models' multi-step reasoning. We apply VCoT to the Visual Storytelling and WikiHow summarization datasets and demonstrate through human evaluation that VCoT offers novel and consistent synthetic data augmentation beating chain-of-thought baselines, which can be used to enhance downstream performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
