视觉-语言模型缺少什么？探究它们在因果顺序推理中的困难
What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning
AI总结：
针对VLM在因果推理中的不足，提出VQA-Causal和VCR-Causal基准，发现模型在因果任务上仅略优于随机猜测，归因于训练数据缺乏因果表达，并通过难负例微调改善。
摘要：
尽管视觉-语言模型（VLM）在下游任务上表现出色，但它们理解和推理视觉输入中因果关系的能力仍不明确。稳健的因果推理是解决复杂高层推理任务的基础，然而现有基准通常包含混合的推理问题，VLM经常可以利用物体识别和活动识别作为捷径来得出正确答案，这使得真正评估其因果推理能力变得困难。为弥补这一差距，我们引入了VQA-Causal和VCR-Causal，这两个新基准专门设计用于隔离和严格评估VLM的因果推理能力。我们的发现表明，虽然VLM在物体和活动识别方面表现出色，但在因果推理任务上表现不佳，通常仅略优于随机猜测。进一步分析表明，这一局限性源于广泛使用的训练数据集中严重缺乏因果表达，其中因果关系很少被明确传达。我们还探索了使用难负例的微调策略，表明有针对性的微调可以在保持泛化和下游性能的同时改善模型的因果推理。我们的研究突出了当前VLM的一个关键差距，并为未来关于因果理解的工作奠定了基础。
英文摘要：
Despite the impressive performance of vision-language models (VLMs) on downstream tasks, their ability to understand and reason about causal relationships in visual inputs remains unclear. Robust causal reasoning is fundamental to solving complex high-level reasoning tasks, yet existing benchmarks often include a mixture of reasoning questions, and VLMs can frequently exploit object recognition and activity identification as shortcuts to arrive at the correct answers, making it challenging to truly assess their causal reasoning abilities. To bridge this gap, we introduce VQA-Causal and VCR-Causal, two new benchmarks specifically designed to isolate and rigorously evaluate VLMs' causal reasoning abilities. Our findings reveal that while VLMs excel in object and activity recognition, they perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. Further analysis suggests that this limitation stems from a severe lack of causal expressions in widely used training datasets, where causal relationships are rarely explicitly conveyed. We additionally explore fine-tuning strategies with hard negative cases, showing that targeted fine-tuning can improve model's causal reasoning while maintaining generalization and downstream performance. Our study highlights a key gap in current VLMs and lays the groundwork for future work on causal understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
