Fact:用忠实、简洁且可迁移的理由教授 MLLM
Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales
AI总结:
本文提出 Fact 范式,通过可验证视觉编程及剪枝、合并、桥接等操作生成忠实、简洁、可迁移的多模态理由,提升不同规模 MLLM 的组合推理、泛化能力并减少幻觉。
摘要:
多模态大语言模型(MLLM)的卓越表现明确证明了它们在处理广泛视觉任务时具备熟练的理解能力。然而,其黑盒推理过程的不透明性质仍是一个难题,使模型缺乏可解释性,并深受幻觉困扰。它们执行复杂组合推理任务的能力也受到限制,最终导致这些模型的学习进程停滞。在本工作中,我们提出 Fact,这是一种新颖范式,旨在生成忠实、简洁且可迁移的多模态理由,用于教授 MLLM。该范式利用可验证的视觉编程生成可执行代码,以保证忠实性和精确性。随后,通过剪枝、合并和桥接等一系列操作,理由的简洁性得到增强。此外,我们筛选能够从编程范式迁移到端到端范式的理由,以保证可迁移性。实验经验证据表明,我们的方法在不同参数规模的模型上均具有优越性,显著提升了它们的组合推理和泛化能力。由于图像与文本之间具有高度相关性,我们的方法还减少了幻觉。
英文摘要:
The remarkable performance of Multimodal Large Language Models (MLLMs) has unequivocally demonstrated their proficient understanding capabilities in handling a wide array of visual tasks. Nevertheless, the opaque nature of their black-box reasoning processes persists as an enigma, rendering them uninterpretable and struggling with hallucination. Their ability to execute intricate compositional reasoning tasks is also constrained, culminating in a stagnation of learning progression for these models. In this work, we introduce Fact, a novel paradigm designed to generate multimodal rationales that are faithful, concise, and transferable for teaching MLLMs. This paradigm utilizes verifiable visual programming to generate executable code guaranteeing faithfulness and precision. Subsequently, through a series of operations including pruning, merging, and bridging, the rationale enhances its conciseness. Furthermore, we filter rationales that can be transferred to end-to-end paradigms from programming paradigms to guarantee transferability. Empirical evidence from experiments demonstrates the superiority of our method across models of varying parameter sizes, significantly enhancing their compositional reasoning and generalization ability. Our approach also reduces hallucinations owing to its high correlation between images and text.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。