arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.15969cs.CVcs.AIcs.CL
收藏

被遗忘的多边形:多模态大语言模型存在形状盲视

Forgotten Polygons: Multimodal Large Language Models are Shape-Blind

Brown University(布朗大学);Tel Aviv University(特拉维夫大学);IIT Kharagpur(印度理工学院卡里帕尔分校);New York University(纽约大学);University of Tübingen(图宾根大学)

William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, Ritambhara Singh

AI总结:

针对多模态大语言模型(MLLMs)在视觉-数学推理上的不足,研究通过评估几何理解、测试多步推理及提出视觉提示思维链(VC-CoT),发现其形状识别缺陷并提升GPT-4o相关任务准确率至93%,揭示系统2推理仍待解决。

摘要:

尽管多模态大语言模型(MLLMs)在视觉-语言任务上表现出色,但它们在数学问题解决方面存在困难,开源模型和最先进模型在视觉-数学基准测试中的表现均低于人类水平。为系统研究MLLMs的视觉-数学推理能力,我们开展了三项工作:(1)评估其对几何基元的理解;(2)测试多步推理能力;(3)探索提升视觉推理能力的潜在方案。研究发现,MLLMs在形状识别方面存在根本性缺陷,顶尖模型识别正多边形的准确率不足50%。通过双过程理论分析这些失败案例,我们发现MLLMs依赖系统1(直觉性、记忆关联)而非系统2(审慎推理)。因此,MLLMs无法对熟悉和新颖形状的边数进行计数,表明它们既未掌握边的概念,也未有效处理视觉输入。最后,我们提出视觉提示思维链(VC-CoT)提示方法,通过明确引用图表中的视觉标注增强多步数学推理能力,将GPT-4o在不规则多边形边数计数任务上的准确率从7%提升至93%。研究结果表明,MLLMs的系统2推理仍是未解决的问题,视觉引导提示对成功激活视觉推理至关重要。代码可在https://github.com/rsinghlab/Shape-Blind获取。

英文摘要:

Despite strong performance on vision-language tasks, Multimodal Large Language Models (MLLMs) struggle with mathematical problem-solving, with both open-source and state-of-the-art models falling short of human performance on visual-math benchmarks. To systematically examine visual-mathematical reasoning in MLLMs, we (1) evaluate their understanding of geometric primitives, (2) test multi-step reasoning, and (3) explore a potential solution to improve visual reasoning capabilities. Our findings reveal fundamental shortcomings in shape recognition, with top models achieving under 50% accuracy in identifying regular polygons. We analyze these failures through the lens of dual-process theory and show that MLLMs rely on System 1 (intuitive, memorized associations) rather than System 2 (deliberate reasoning). Consequently, MLLMs fail to count the sides of both familiar and novel shapes, suggesting they have neither learned the concept of sides nor effectively process visual inputs. Finally, we propose Visually Cued Chain-of-Thought (VC-CoT) prompting, which enhances multi-step mathematical reasoning by explicitly referencing visual annotations in diagrams, boosting GPT-4o's accuracy on an irregular polygon side-counting task from 7% to 93%. Our findings suggest that System 2 reasoning in MLLMs remains an open problem, and visually-guided prompting is essential for successfully engaging visual reasoning. Code available at: https://github.com/rsinghlab/Shape-Blind.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑