arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.16385cs.CVcs.CL
收藏

逐步合成:工具、模板与LLM作为基于推理的图表VQA数据生成器

Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA

Johns Hopkins University(约翰斯·霍普金斯大学);AWS AI Labs(亚马逊云科技人工智能实验室)

Zhuowan Li, Bhavan Jasani, Peng Tang, Shabnam Ghadar

AI总结:

本文提出逐步合成策略,利用LLM将复杂图表问题分解为子问题并结合Python工具生成增强数据,显著提升ChartQA和PlotQA上的推理问答准确率。

摘要:

理解图表和绘图等数据可视化需要对视觉元素和数值进行推理。尽管当前的图表视觉问答(chart VQA)模型在抽取式问题上表现强劲,但在复杂推理问题上表现不佳。本文通过数据增强来解决推理能力不足的问题。我们利用已展现出强大推理能力的大语言模型(LLMs)作为自动数据标注器,为图表图像生成问答标注。我们方法的关键创新在于“逐步合成”(Synthesize Step-by-Step)策略:基于LLM的数据生成器学习将复杂问题分解为逐步的子问题(推理依据),然后使用外部工具(即Python)推导出最终答案。这一逐步生成过程在基于模板的问答生成流水线所生成的合成数据上进行训练。实验结果凸显了所提出的逐步生成方法的重要性。通过使用LLM增强数据(LAMENDA)进行训练,我们显著提升了图表VQA模型,在ChartQA和PlotQA数据集上取得了最先进的准确率。特别是,我们的方法将先前最先进方法在ChartQA数据集中人工编写问题上的准确率从38%提高到54%,这些问题需要较强的推理能力。我们希望我们的工作能够强调合成数据的潜力,并鼓励进一步探索利用LLM进行数据增强以应对推理密集型任务。

英文摘要:

Understanding data visualizations like charts and plots requires reasoning about both visual elements and numerics. Although strong in extractive questions, current chart visual question answering (chart VQA) models suffer on complex reasoning questions. In this work, we address the lack of reasoning ability by data augmentation. We leverage Large Language Models (LLMs), which have shown to have strong reasoning ability, as an automatic data annotator that generates question-answer annotations for chart images. The key innovation in our method lies in the Synthesize Step-by-Step strategy: our LLM-based data generator learns to decompose the complex question into step-by-step sub-questions (rationales), which are then used to derive the final answer using external tools, i.e. Python. This step-wise generation procedure is trained on synthetic data generated using a template-based QA generation pipeline. Experimental results highlight the significance of the proposed step-by-step generation. By training with the LLM-augmented data (LAMENDA), we significantly enhance the chart VQA models, achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets. In particular, our approach improves the accuracy of the previous state-of-the-art approach from 38% to 54% on the human-written questions in the ChartQA dataset, which needs strong reasoning. We hope our work underscores the potential of synthetic data and encourages further exploration of data augmentation using LLMs for reasoning-heavy tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑