ChartInstruct：面向图表理解与推理的指令微调
ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning
AI总结：
本文提出ChartInstruct，一种基于71K图表生成191K指令的图表专用指令微调数据集，并设计了端到端和流水线两种系统，在四个下游任务上取得最先进结果，提升了图表理解与推理的通用性。
摘要：
图表提供了数据的可视化表示，被广泛用于分析信息、解答疑问以及向他人传达见解。近年来，出现了各种与图表相关的下游任务，例如问答和摘要。解决这些任务的一种常见策略是对最初在视觉任务语言上训练的各种模型进行微调。然而，这种特定于任务的模型无法解决广泛的图表相关任务，限制了其在实际应用中的适用性。为了克服这些挑战，我们引入了ChartInstruct：一个新颖的、特定于图表的视觉语言指令遵循数据集，包含由71K个图表生成的191K条指令。然后，我们提出了两种不同的系统用于此类数据集的指令微调：（1）一种端到端模型，将用于图表理解的视觉编码器与大型语言模型（LLM）连接起来；（2）一种流水线模型，采用两步方法提取图表数据表并将其输入到LLM中。在四个下游任务的实验中，我们首先展示了我们模型的有效性——取得了一系列新的最先进结果。进一步的评估表明，我们的指令微调方法支持广泛的真实世界图表理解和推理场景，从而将我们模型的范围和适用性扩展到新类型的任务。
英文摘要：
Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chart-related tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
