arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.16420cs.CVcs.CL
收藏

超越端到端 VLMs:利用中间文本表示实现更优的流程图理解

Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding

New Jersey Institute of Technology(新泽西理工学院);Pennsylvania State University(宾夕法尼亚州立大学)

Junyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang

AI总结:

针对端到端VLMs在流程图理解中可控性和可解释性不足的问题,提出两阶段框架TextFlow,通过中间文本表示提升性能、可控性与可解释性,在FlowVQA和FlowLearn基准上达到SOTA。

摘要:

流程图通常以图像形式呈现,这推动了使用视觉语言模型(VLMs)进行端到端流程图理解的趋势。然而,这带来了两个关键挑战:(i) 有限的 controllability——用户对下游任务的影响极小,因为他们只能修改输入图像,而 VLMs 的训练对大多数研究人员来说往往难以触及。(ii) 缺乏 explainability——很难将 VLM 的错误追溯到具体原因,例如视觉编码或推理中的失败。我们提出了 TextFlow,通过两个阶段解决上述问题:(i) Vision Textualizer——从流程图图像生成文本表示;(ii) Textual Reasoner——基于文本表示进行问答。TextFlow 提供了三个关键优势:(i) 用户可以选择文本表示的类型(例如 Graphviz、Mermaid、PlantUML),或进一步将其转换为可执行的图对象以调用工具,从而提升性能和 controllability;(ii) 它通过帮助更清晰地将错误归因于视觉或文本处理组件来改善 explainability;(iii) 它促进了解决方案的模块化,例如当 VLMs 在端到端方式下表现不佳时,允许在 Reasoner 阶段使用高级 LLMs。在 FlowVQA 和 FlowLearn 基准上的实验证明了 TextFlow 的最先进性能及其鲁棒性。所有代码均已公开。

英文摘要:

Flowcharts are typically presented as images, driving the trend of using vision-language models (VLMs) for end-to-end flowchart understanding. However, two key challenges arise: (i) Limited controllability--users have minimal influence over the downstream task, as they can only modify input images, while the training of VLMs is often out of reach for most researchers. (ii) Lack of explainability--it is difficult to trace VLM errors to specific causes, such as failures in visual encoding or reasoning. We propose TextFlow, addressing aforementioned issues with two stages: (i) Vision Textualizer--which generates textual representations from flowchart images; and (ii) Textual Reasoner--which performs question-answering based on the text representations. TextFlow offers three key advantages: (i) users can select the type of text representations (e.g., Graphviz, Mermaid, PlantUML), or further convert them into executable graph object to call tools, enhancing performance and controllability; (ii) it improves explainability by helping to attribute errors more clearly to visual or textual processing components; and (iii) it promotes the modularization of the solution, such as allowing advanced LLMs to be used in the Reasoner stage when VLMs underperform in end-to-end fashion. Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow's state-of-the-art performance as well as its robustness. All code is publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑