KAM-CoT：知识增强的多模态思维链推理
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
AI总结：
本文提出KAM-CoT框架，融合知识图谱、多模态信息与两阶段思维链训练，以较低参数规模在ScienceQA上显著超越GPT-3.5和GPT-4。
摘要：
大语言模型（LLM）通过利用能够实现逐步思考的思维链（CoT），在自然语言处理任务中展现了令人瞩目的性能。近期，为LLM扩展多模态能力成为研究热点，但这会带来计算开销，并需要大量硬件资源。为应对这些挑战，我们提出KAM-CoT，这是一个整合CoT推理、知识图谱（KG）和多种模态的框架，以全面理解多模态任务。KAM-CoT采用带有KG接地的两阶段训练流程，以生成有效的推理依据和答案。通过在推理过程中融入来自KG的外部知识，模型获得更深入的上下文理解，从而减少幻觉并提升答案质量。这种知识增强的CoT推理使模型能够处理需要外部上下文的问题，并给出更有依据的答案。实验结果表明，KAM-CoT优于最先进方法。在ScienceQA数据集上，我们取得93.87%的平均准确率，分别比GPT-3.5（75.17%）高18%、比GPT-4（83.99%）高10%。值得注意的是，KAM-CoT一次仅使用280M个可训练参数便取得这些结果，证明了其成本效率和有效性。
英文摘要：
Large Language Models (LLMs) have demonstrated impressive performance in natural language processing tasks by leveraging chain of thought (CoT) that enables step-by-step thinking. Extending LLMs with multimodal capabilities is the recent interest, but incurs computational cost and requires substantial hardware resources. To address these challenges, we propose KAM-CoT a framework that integrates CoT reasoning, Knowledge Graphs (KGs), and multiple modalities for a comprehensive understanding of multimodal tasks. KAM-CoT adopts a two-stage training process with KG grounding to generate effective rationales and answers. By incorporating external knowledge from KGs during reasoning, the model gains a deeper contextual understanding reducing hallucinations and enhancing the quality of answers. This knowledge-augmented CoT reasoning empowers the model to handle questions requiring external context, providing more informed answers. Experimental findings show KAM-CoT outperforms the state-of-the-art methods. On the ScienceQA dataset, we achieve an average accuracy of 93.87%, surpassing GPT-3.5 (75.17%) by 18% and GPT-4 (83.99%) by 10%. Remarkably, KAM-CoT achieves these results with only 280M trainable parameters at a time, demonstrating its cost-efficiency and effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
