面向语言模型思维链推理的多模态隐空间学习
Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models
AI总结：
针对现有多模态思维链中视觉特征与语言思维对齐不足的问题，提出基于扩散过程隐空间学习的多模态CoT推理方法，深层融合图文特征，在ScienceQA基准上取得SOTA性能，提升语言模型多模态复杂推理能力。
摘要：
思维链（Chain-of-thought, CoT）推理已在语言模型解决复杂任务、回答问题的场景中展现出优异性能。但现实世界的诸多问题需要文本、图像等多模态信息支撑。过往多模态思维链的研究主要聚焦于从现成视觉模型中提取固定图像特征，再通过注意力机制将其与文本融合。该方法存在局限性：这些视觉模型并非为复杂推理任务设计，与语言思维的对齐程度不足。为突破这一局限，我们提出一种全新的多模态思维链推理方法，通过扩散过程进行隐空间学习，生成与语言思维对齐的有效图像特征。本方法在深层维度融合图像特征与文本表示，提升多模态思维链的复杂推理能力。我们在多模态ScienceQA（科学问答）基准与机器翻译基准上验证了方法的有效性，在ScienceQA上取得了当前最优（state-of-the-art）性能。总体而言，该方法为语言模型的多模态推理提供了更鲁棒、更有效的解决方案，增强了其应对复杂现实问题的能力。
英文摘要：
Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. However, many real-world questions require multi-modal information, such as text and images. Previous research on multi-modal CoT has primarily focused on extracting fixed image features from off-the-shelf vision models and then fusing them with text using attention mechanisms. This approach has limitations because these vision models were not designed for complex reasoning tasks and do not align well with language thoughts. To overcome this limitation, we introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT. We demonstrate the efficacy of our proposed method on multi-modal ScienceQA and machine translation benchmarks, achieving state-of-the-art performance on ScienceQA. Overall, our approach offers a more robust and effective solution for multi-modal reasoning in language models, enhancing their ability to tackle complex real-world problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
