arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.18641cs.CLcs.CV
收藏

通过图表-表格对的跨模态预训练增强视觉与语言任务中的图表理解

Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairs

Columbia University(哥伦比亚大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virginia Tech(弗吉尼亚理工大学)

Mingyang Zhou, Yi R. Fung, Long Chen, Christopher Thomas, Heng Ji, Shih-Fu Chang

AI总结:

本文提出ChartT5模型,通过在图表-表格对上设计掩码表头预测和掩码数值预测两种跨模态预训练目标,显著提升了图表问答与摘要性能,在ChartQA上超越非预训练方法8%以上。

摘要:

构建能够理解图表并传达其背后显著信息的跨模型智能,是视觉与语言(V+L)领域一个极具吸引力的挑战。揭示图表图形所对应的底层表格数据的能力,是实现自动图表理解的关键。我们提出了ChartT5,这是一种V+L模型,通过在图表-表格对上进行跨模态预训练,学习如何从图表图像中解读表格信息。具体来说,我们提出了两个新颖的预训练目标:掩码表头预测(MHP)和掩码数值预测(MVP),以赋予模型解读表格信息的不同技能。我们在图表问答和图表摘要任务上进行了大量实验,以验证所提预训练策略的有效性。特别是在ChartQA基准上,我们的ChartT5以超过8%的性能提升超越了当前最先进的非预训练方法。

英文摘要:

Building cross-model intelligence that can understand charts and communicate the salient information hidden behind them is an appealing challenge in the vision and language(V+L) community. The capability to uncover the underlined table data of chart figures is a critical key to automatic chart understanding. We introduce ChartT5, a V+L model that learns how to interpret table information from chart images via cross-modal pre-training on plot table pairs. Specifically, we propose two novel pre-training objectives: Masked Header Prediction (MHP) and Masked Value Prediction (MVP) to facilitate the model with different skills to interpret the table information. We have conducted extensive experiments on chart question answering and chart summarization to verify the effectiveness of the proposed pre-training strategies. In particular, on the ChartQA benchmark, our ChartT5 outperforms the state-of-the-art non-pretraining methods by over 8% performance gains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑