ChartGemma：面向真实场景图表推理的视觉指令微调
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
AI总结：
针对现有图表推理模型依赖底层数据表、泛化性弱的问题，研究提出基于PaliGemma的ChartGemma模型，通过图表图像生成的指令数据训练，在5个基准上取得最优效果并开源相关资源。
摘要：
鉴于图表作为数据分析、可视化与决策工具在各行业和科学领域的普及，人们对开发用于图表理解与推理的预训练基础模型以及通用指令微调模型的兴趣日益浓厚。然而，现有方法在影响图表表示模型性能的两个关键维度上存在严重缺陷：它们基于图表的底层数据表生成的数据进行训练，忽略了图表图像中的视觉趋势和模式，并且使用弱对齐的视觉-语言骨干模型进行领域特定训练，限制了其在真实场景中遇到图表时的泛化能力。我们针对这些重要缺陷进行改进，提出了ChartGemma——一种基于PaliGemma开发的新型图表理解与推理模型。ChartGemma不依赖底层数据表，而是基于直接从图表图像生成的指令微调数据进行训练，从而能够从多样化的图表中捕捉高层趋势和底层视觉信息。我们的简单方法在涵盖图表摘要生成、问答和事实验证的5个基准测试上取得了最优结果，并且我们对真实世界图表开展的细致定性研究表明，与同类模型相比，ChartGemma生成的摘要更符合实际且事实准确性更高。我们在https://github.com/vis-nlp/ChartGemma 上发布了代码、模型检查点、数据集和演示。
英文摘要：
Given the ubiquity of charts as a data analysis, visualization, and decision-making tool across industries and sciences, there has been a growing interest in developing pre-trained foundation models as well as general purpose instruction-tuned models for chart understanding and reasoning. However, existing methods suffer crucial drawbacks across two critical axes affecting the performance of chart representation models: they are trained on data generated from underlying data tables of the charts, ignoring the visual trends and patterns in chart images, and use weakly aligned vision-language backbone models for domain-specific training, limiting their generalizability when encountering charts in the wild. We address these important drawbacks and introduce ChartGemma, a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables, ChartGemma is trained on instruction-tuning data generated directly from chart images, thus capturing both high-level trends and low-level visual information from a diverse set of charts. Our simple approach achieves state-of-the-art results across $5$ benchmarks spanning chart summarization, question answering, and fact-checking, and our elaborate qualitative studies on real-world charts show that ChartGemma generates more realistic and factually correct summaries compared to its contemporaries. We release the code, model checkpoints, dataset, and demos at https://github.com/vis-nlp/ChartGemma.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
