arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.02173cs.CVcs.AIcs.MM
收藏

ChartReader:一个无需启发式规则的图表反渲染与理解统一框架

ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules

Carnegie Mellon University(卡内基梅隆大学);Microsoft Research(微软研究院);Amazon(亚马逊)

Zhi-Qi Cheng, Qi Dai, Siyao Li, Jingdong Sun, Teruko Mitamura, Alexander G. Hauptmann

AI总结:

提出ChartReader统一框架,通过基于transformer的组件检测和扩展预训练视觉语言模型,实现无需启发式规则的图表反渲染与理解,在多项任务上优于现有方法,并支持与主流LLM即插即用集成。

摘要:

图表是直观传达复杂数据的强大工具,但由于图表类型多样且组件复杂,其理解面临挑战。现有图表理解方法要么依赖启发式规则,要么过度依赖OCR系统,导致性能欠佳。为解决这些问题,我们提出了ChartReader,一个无缝集成图表反渲染与理解任务的统一框架。该方法包含一个基于transformer的图表组件检测模块,以及一个扩展的预训练视觉语言模型,用于图表到X任务。通过从标注数据集中自动学习图表规则,该方法无需手动制定规则,减少了工作量并提高了准确性。我们还引入了数据变量替换技术,并扩展了预训练模型的输入和位置嵌入以支持跨任务训练。我们在Chart-to-Table、ChartQA和Chart-to-Text任务上评估了ChartReader,证明其优于现有方法。该框架可显著减少图表分析中的人工投入,向通用图表理解模型迈进一步。此外,该方法可与T5和TaPas等主流LLM即插即用地集成,扩展其图表理解能力。代码可在https://github.com/zhiqic/ChartReader获取。

英文摘要:

Charts are a powerful tool for visually conveying complex data, but their comprehension poses a challenge due to the diverse chart types and intricate components. Existing chart comprehension methods suffer from either heuristic rules or an over-reliance on OCR systems, resulting in suboptimal performance. To address these issues, we present ChartReader, a unified framework that seamlessly integrates chart derendering and comprehension tasks. Our approach includes a transformer-based chart component detection module and an extended pre-trained vision-language model for chart-to-X tasks. By learning the rules of charts automatically from annotated datasets, our approach eliminates the need for manual rule-making, reducing effort and enhancing accuracy.~We also introduce a data variable replacement technique and extend the input and position embeddings of the pre-trained model for cross-task training. We evaluate ChartReader on Chart-to-Table, ChartQA, and Chart-to-Text tasks, demonstrating its superiority over existing methods. Our proposed framework can significantly reduce the manual effort involved in chart analysis, providing a step towards a universal chart understanding model. Moreover, our approach offers opportunities for plug-and-play integration with mainstream LLMs such as T5 and TaPas, extending their capability to chart comprehension tasks. The code is available at https://github.com/zhiqic/ChartReader.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑