ChartLlama：用于图表理解与生成的多模态大语言模型
ChartLlama: A Multimodal LLM for Chart Understanding and Generation
AI总结：
本文提出ChartLlama，通过GPT-4构建多步骤指令微调数据集，训练多模态大语言模型，在图表问答、图表转文本和图表提取等基准上超越现有方法，验证了该数据生成方法对提升图表理解能力的价值。
摘要：
多模态大语言模型在大多数视觉语言任务上已展现出令人印象深刻的表现。然而，这类模型通常缺乏对特定领域数据的理解能力，尤其是在解读图表图像时。这主要是由于缺乏相关的多模态指令微调数据集。在本文中，我们利用GPT-4创建了一个高质量的指令微调数据集。我们开发了一个多步骤数据生成流程，其中不同步骤分别负责生成表格数据、创建图表图像以及设计指令微调数据。我们方法的灵活性使我们能够持续高效地生成多样化、高质量的指令微调数据，同时保持较低的资源消耗。此外，它还使我们能够纳入现有数据集中尚未涵盖的更广泛的图表类型和任务类型。接下来，我们介绍了ChartLlama，这是一个使用我们所创建的数据集训练的多模态大语言模型。ChartLlama在ChartQA、Chart-to-text和Chart-extraction评估基准上优于所有先前方法。此外，在我们专门编制的包含新图表类型和任务类型的图表数据集上，ChartLlama相较于基线模型取得了显著提升。ChartLlama的结果证实了我们所提出的数据生成方法在增强图表理解方面的价值和巨大潜力。
英文摘要：
Multi-modal large language models have demonstrated impressive performances on most vision-language tasks. However, the model generally lacks the understanding capabilities for specific domain data, particularly when it comes to interpreting chart figures. This is mainly due to the lack of relevant multi-modal instruction tuning datasets. In this article, we create a high-quality instruction-tuning dataset leveraging GPT-4. We develop a multi-step data generation process in which different steps are responsible for generating tabular data, creating chart figures, and designing instruction tuning data separately. Our method's flexibility enables us to generate diverse, high-quality instruction-tuning data consistently and efficiently while maintaining a low resource expenditure. Additionally, it allows us to incorporate a wider variety of chart and task types not yet featured in existing datasets. Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset. ChartLlama outperforms all prior methods in ChartQA, Chart-to-text, and Chart-extraction evaluation benchmarks. Additionally, ChartLlama significantly improves upon the baseline in our specially compiled chart dataset, which includes new chart and task types. The results of ChartLlama confirm the value and huge potential of our proposed data generation method in enhancing chart comprehension.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
