语言模型中的多模态思维链推理
Multimodal Chain-of-Thought Reasoning in Language Models
AI总结：
针对现有思维链研究多聚焦语言模态的问题，提出整合文本与视觉模态的两阶段多模态思维链框架，在ScienceQA等基准上实现了10亿参数以下模型的最优性能，还可缓解幻觉、加快收敛。
摘要：
大语言模型（LLM）通过利用思维链（CoT）提示生成中间推理链作为推断答案的依据，已在复杂推理任务中展现出出色性能。然而现有CoT研究主要聚焦于语言模态。我们提出Multimodal-CoT（多模态思维链），将语言（文本）和视觉（图像）模态整合到一个两阶段框架中，将依据生成与答案推断分离。如此一来，答案推断便可利用基于多模态信息生成的更优质依据。在ScienceQA和A-OKVQA基准数据集上的实验结果验证了所提方法的有效性。借助Multimodal-CoT，我们参数规模不足10亿的模型在ScienceQA基准上取得了当前最优性能。分析表明，Multimodal-CoT具备缓解幻觉问题、提升收敛速度的优势。相关代码已公开于https://github.com/amazon-science/mm-cot。
英文摘要：
Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
