BBA:面向大视觉-语言模型推理的双模态行为对齐
BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models
AI总结:
本文提出双模态行为对齐(BBA)提示方法,通过分别生成并对齐视觉与DSL推理链,提升大视觉-语言模型在几何、国际象棋和分子预测等复杂多模态推理任务中的表现。
摘要:
多模态推理是大视觉-语言模型(LVLMs)的一项关键能力。与提供精确视觉表示的领域特定语言(DSL)相结合,使这些模型有机会在复杂且专业的领域中执行更准确的推理。然而,普通的思维链(CoT)提示方法在有效利用视觉表示和DSL表示的独特优势方面面临挑战,这主要是由于它们推理机制不同。此外,它在多步推理任务中往往无法充分处理关键步骤。为缓解这些挑战,我们提出了双模态行为对齐(BBA)提示方法,旨在最大化DSL在增强复杂多模态推理任务中的潜力。该方法首先引导LVLMs为视觉表示和DSL表示分别创建推理链。随后,它通过处理任何不一致之处来对齐这些推理链,从而实现来自不同模态行为的协调整合。我们的实验表明,BBA显著提升了GPT-4V(ision)在几何问题求解(28.34% → 34.22%)、国际象棋局面优势预测(42.08% → 46.99%)和分子性质预测(77.47% → 83.52%)上的性能。
英文摘要:
Multimodal reasoning stands as a pivotal capability for large vision-language models (LVLMs). The integration with Domain-Specific Languages (DSL), offering precise visual representations, equips these models with the opportunity to execute more accurate reasoning in complex and professional domains. However, the vanilla Chain-of-Thought (CoT) prompting method faces challenges in effectively leveraging the unique strengths of visual and DSL representations, primarily due to their differing reasoning mechanisms. Additionally, it often falls short in addressing critical steps in multi-step reasoning tasks. To mitigate these challenges, we introduce the \underline{B}i-Modal \underline{B}ehavioral \underline{A}lignment (BBA) prompting method, designed to maximize the potential of DSL in augmenting complex multi-modal reasoning tasks. This method initiates by guiding LVLMs to create separate reasoning chains for visual and DSL representations. Subsequently, it aligns these chains by addressing any inconsistencies, thus achieving a cohesive integration of behaviors from different modalities. Our experiments demonstrate that BBA substantially improves the performance of GPT-4V(ision) on geometry problem solving ($28.34\% \to 34.22\%$), chess positional advantage prediction ($42.08\% \to 46.99\%$) and molecular property prediction ($77.47\% \to 83.52\%$).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。