arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.14928cs.CL
收藏

过程链:用于过程问答的层次视觉语言推理

Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA

NLP;CT Lab, Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门大学CT实验室);Department of Electrical and Computer Engineering, University of Macau(电气与计算机工程系,澳门大学);Centre for Cognitive and Brain Sciences, University of Macau(认知与脑科学中心,澳门大学)

Guanhua Chen, Yutong Yao, Shenghe Sun, Ci-Jun Gao, Shudong Liu, Lidia S. Chao, Feng Wan, Derek F. Wong

AI总结:

本文提出ProcedureVQA基准,针对视觉过程问答任务,通过过程链框架解决现有VLMs在跨模态检索和步骤分解上的不足,实验显示其在六个VLMs上提升达13%。

摘要:

近年来,视觉语言模型(VLMs)在标准图像-文本任务上取得了显著成果,但其在视觉过程问答(VP-QA)中的潜力仍待探索。VP-QA面临独特挑战,用户通过上传图像查询复杂过程的下一步动作。为系统评估VLMs在该任务上的表现,我们提出了ProcedureVQA,一个专门用于视觉过程推理的多模态基准。通过深入分析,我们发现当前VLMs存在两个关键局限:在给定视觉状态时无法有效检索结构化过程,以及图像序列粒度与文本步骤分解不匹配。为解决这些问题,我们提出了过程链(CoP),一种层次推理框架,首先利用视觉线索检索相关指令,然后通过语义分解进行步骤细化,最后生成下一步。在六个VLMs上的实验表明,CoP的有效性显著,相对于标准基线,最大提升达13%。

英文摘要:

Recent advances in vision-language models (VLMs) have achieved impressive results on standard image-text tasks, yet their potential for visual procedure question answering (VP-QA) remains largely unexplored. VP-QA presents unique challenges where users query next-step actions by uploading images for intermediate states of complex procedures. To systematically evaluate VLMs on this practical task, we propose ProcedureVQA, a novel multimodal benchmark specifically designed for visual procedural reasoning. Through comprehensive analysis, we identify two critical limitations in current VLMs: inadequate cross-modal retrieval of structured procedures given visual states, and misalignment between image sequence granularity and textual step decomposition. To address these issues, we present Chain-of-Procedure (CoP), a hierarchical reasoning framework that first retrieves relevant instructions using visual cues, then performs step refinement through semantic decomposition, and finally generates the next step. Experiments across six VLMs demonstrate CoP's effectiveness, achieving up to 13% absolute improvement over standard baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑