TinyChart:基于视觉Token合并与Program-of-Thoughts学习的高效图表理解
TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning
AI总结:
针对多模态大模型计算成本高的问题,提出仅3B参数的TinyChart,通过视觉Token合并模块与Program-of-Thoughts学习策略分别解决高分辨率图像特征冗余和数值计算负担,在多项图表理解基准上达到SOTA并实现高效推理。
摘要:
图表对于呈现和解释复杂数据关系至关重要。近期,多模态大语言模型(MLLMs)在各种图表理解任务中展现出卓越能力。然而,这些模型在参数量和计算需求上的庞大规模限制了其在资源受限环境中的使用。在本文中,我们提出了TinyChart,一个仅有3B参数的高效图表理解MLLM。TinyChart克服了高效图表理解中的两个关键挑战:(1)通过Program-of-Thoughts(PoT)学习策略减轻学习数值计算的负担,该策略训练模型生成Python程序进行数值计算;(2)通过Vision Token Merging模块减少视觉Transformer为高分辨率图像产生的冗长视觉特征序列,该模块逐步合并最相似的视觉token。大量实验表明,我们的3B TinyChart在ChartQA、Chart-to-Text、Chart-to-Table、OpenCQA和ChartX等多种图表理解基准上实现了SOTA性能。它超越了ChartLlama和ChartAst等参数量高达13B的多个图表理解MLLM,以及在ChartQA上超越闭源通用MLLM GPT-4V。由于更小的模型规模和更高效的视觉编码,它在推理期间展现出更高的吞吐量,证明了其卓越的效率。我们的代码和模型可在 https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/TinyChart 获取。
英文摘要:
Charts are important for presenting and explaining complex data relationships. Recently, multimodal large language models (MLLMs) have shown remarkable capabilities in various chart understanding tasks. However, the sheer size of these models in terms of parameters and computational requirements limits their use in resource-constrained environments. In this paper, we present TinyChart, an efficient MLLM for chart understanding with only 3B parameters. TinyChart overcomes two key challenges in efficient chart understanding: (1) reduce the burden of learning numerical computations through a Program-of-Thoughts (PoT) learning strategy, which trains the model to generate Python programs for numerical calculations, and (2) reduce lengthy vision feature sequences produced by the vision transformer for high-resolution images through a Vision Token Merging module, which gradually merges most similar vision tokens. Extensive experiments demonstrate that our 3B TinyChart achieves SOTA performance on a variety of chart understanding benchmarks including ChartQA, Chart-to-Text, Chart-to-Table, OpenCQA, and ChartX. It outperforms several chart understanding MLLM with up to 13B parameters such as ChartLlama and ChartAst, and close-sourced general-purpose MLLM GPT-4V on ChartQA. It also demonstrates its superior efficiency with higher throughput during inference due to a smaller model scale and more efficient vision encoding. Our code and model are available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/TinyChart.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。