FlowLearn:评估大视觉语言模型的流程图理解能力
FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding
AI总结:
本研究推出包含科学与模拟两类流程图的FlowLearn数据集,配套多类标注,用于评估大视觉语言模型的流程图理解能力,测试发现现有模型各有优劣,无模型能在所有任务中领先,为该领域后续发展指明方向。
摘要:
流程图是用简洁视觉表征呈现复杂概念的图形工具。本文推出FlowLearn数据集,这是专门为提升流程图理解能力构建的资源。FlowLearn包含复杂科学流程图和模拟流程图两类:科学子数据集含3858张来自科学文献的流程图,模拟子数据集含10000张通过可定制脚本生成的流程图。该数据集还补充了视觉组件标注、OCR(光学字符识别)结果、Mermaid代码表征以及VQA(视觉问答)问答对。尽管大视觉语言模型(LVLMs)已在各类视觉理解任务中展现出可靠能力,但它们在解析流程图这一科学交流关键元素上的效果仍未得到充分研究。FlowLearn测试集专为评估LVLMs的流程图理解性能设计。本研究对前沿LVLMs开展了全面评估,明确了现有局限性,为这一相对未被充分探索领域的后续优化奠定了基础。例如在模拟流程图相关任务中,GPT-4V在节点计数任务上取得最高准确率(58%),而Claude在OCR任务上取得最高准确率(83%)。值得注意的是,没有任何一个模型能在FlowLearn框架的所有任务中都表现优异,这凸显了该领域仍有很大的进一步发展空间。
英文摘要:
Flowcharts are graphical tools for representing complex concepts in concise visual representations. This paper introduces the FlowLearn dataset, a resource tailored to enhance the understanding of flowcharts. FlowLearn contains complex scientific flowcharts and simulated flowcharts. The scientific subset contains 3,858 flowcharts sourced from scientific literature and the simulated subset contains 10,000 flowcharts created using a customizable script. The dataset is enriched with annotations for visual components, OCR, Mermaid code representation, and VQA question-answer pairs. Despite the proven capabilities of Large Vision-Language Models (LVLMs) in various visual understanding tasks, their effectiveness in decoding flowcharts - a crucial element of scientific communication - has yet to be thoroughly investigated. The FlowLearn test set is crafted to assess the performance of LVLMs in flowchart comprehension. Our study thoroughly evaluates state-of-the-art LVLMs, identifying existing limitations and establishing a foundation for future enhancements in this relatively underexplored domain. For instance, in tasks involving simulated flowcharts, GPT-4V achieved the highest accuracy (58%) in counting the number of nodes, while Claude recorded the highest accuracy (83%) in OCR tasks. Notably, no single model excels in all tasks within the FlowLearn framework, highlighting significant opportunities for further development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。