arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10085cs.CL
收藏

通过有效的预训练任务增强图表问答能力

Enhancing Question Answering on Charts Through Effective Pre-training Tasks

University of Utah(犹他大学);University of Pennsylvania(宾夕法尼亚大学);Bloomberg(彭博公司)

Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang, Shalin Shah

AI总结:

本文针对现有VisualQA模型在图表问答中的不足,提出三种预训练任务以增强模型的结构-视觉知识与数值理解能力,并在三个图表数据集上验证MatCha-v2模型平均提升1.7%。

摘要:

要完全理解一份文档,仅使用文本信息是不够的,还需要理解布局和图表等视觉线索。虽然当前最先进的文档理解方法(基于OCR和无OCR的方法)表现良好,但尚未对其能力和局限性进行彻底分析。因此,在这项工作中,我们针对当前VisualQA模型在应用于图表和图形时的局限性进行了研究。为了探究最先进模型的不足,我们以ChartQA作为案例研究,进行了全面的行为分析。我们的发现表明,现有模型在回答与图表的结构和视觉上下文以及数值信息相关的问题时表现尤其不佳。为了解决这些问题,我们提出了三个简单的预训练任务,以增强现有模型在结构-视觉知识以及数值问题理解方面的能力。我们在三个图表数据集上(包括抽取式和生成式问答数据集)评估了我们的预训练模型(称为MatCha-v2),并观察到其相比基线模型平均提升了1.7%。

英文摘要:

To completely understand a document, the use of textual information is not enough. Understanding visual cues, such as layouts and charts, is also required. While the current state-of-the-art approaches for document understanding (both OCR-based and OCR-free) work well, a thorough analysis of their capabilities and limitations has not yet been performed. Therefore, in this work, we addresses the limitation of current VisualQA models when applied to charts and plots. To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study. Our findings indicate that existing models particularly underperform in answering questions related to the chart's structural and visual context, as well as numerical information. To address these issues, we propose three simple pre-training tasks that enforce the existing model in terms of both structural-visual knowledge, as well as its understanding of numerical questions. We evaluate our pre-trained model (called MatCha-v2) on three chart datasets - both extractive and abstractive question datasets - and observe that it achieves an average improvement of 1.7% over the baseline model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑