COCO-Tree：用于增强视觉语言模型推理能力的组合层次概念树
COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models
AI总结：
本文提出COCO-Tree方法，通过从LLMs学习神经符号概念树来增强VLMs的组合推理能力，在四个基准上使七种开源VLMs的组合泛化提升5-10%，并提供可解释的推理过程。
摘要：
组合推理仍然是现代视觉语言模型（VLMs）的一个持续弱点：当任务取决于理解图像中多个对象、属性和关系如何交互时，它们常常会出错。多项研究工作试图通过改进提示结构、思维链推理等创新技巧来提升组合性表现。最近的一些工作尝试利用训练良好的大型语言模型（LLMs）为VLMs赋予额外的推理能力，由于LLMs在语言理解方面远优于VLMs，以此来弥补VLMs有限的语言能力。然而，这些方法要么资源消耗大，要么无法提供可解释的推理过程。在本文中，我们提出了“COCO-Tree”——一种新颖的方法，通过用从LLMs学习到的精心设计的神经符号概念树来增强VLM输出，从而提升VLM的语言推理能力。COCO-Tree受束搜索启发的推理过程不仅提升了组合性表现，还为VLM的预测提供了背后的逻辑依据。在Winoground、EqBench、ColorSwap和SugarCrepe四个组合性基准上，对七种不同规模的开源VLMs进行的实验结果表明，COCO-Tree相较于基线方法显著提升了5-10%的组合泛化能力。
英文摘要：
Compositional reasoning remains a persistent weakness of modern vision language models (VLMs): they often falter when a task hinges on understanding how multiple objects, attributes, and relations interact within an image. Multiple research works have attempted to improve compositionality performance by creative tricks such as improving prompt structure, chain of thought reasoning, etc. A more recent line of work attempts to impart additional reasoning in VLMs using well-trained Large Language Models (LLMs), which are far superior in linguistic understanding than VLMs to compensate for the limited linguistic prowess of VLMs. However, these approaches are either resource-intensive or do not provide an interpretable reasoning process. In this paper, we present 'COCO-Tree' - a novel approach that augments VLM outputs with carefully designed neurosymbolic concept trees learned from LLMs to improve VLM's linguistic reasoning. COCO-Tree's beam search-inspired reasoning process boosts compositionality performance and provides a rationale behind VLM predictions. Empirical results on four compositionality benchmarks, Winoground, EqBench, ColorSwap, and SugarCrepe, in seven different open-source VLMs with varying sizes, demonstrate that COCO-Tree significantly improves compositional generalization by 5-10% over baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
