START: 空间与文本学习用于图表理解
START: Spatial and Textual Learning for Chart Understanding
AI总结：
START通过空间与文本学习提升图表理解能力，引入图表元素定位和图表到代码生成，增强多模态大语言模型对图表结构和数据细节的理解。
摘要：
图表理解对于在现实场景中部署多模态大语言模型（MLLMs）至关重要，例如分析科学论文和技术报告。与自然图像不同，图表结合了结构化的视觉布局（空间属性）和底层数据表示（文本属性）——把握两者对于精确、细粒度的图表推理至关重要。受此观察的启发，我们提出了START，即用于图表理解的空间与文本学习。具体而言，我们引入了（i）图表元素定位和（ii）图表到代码生成，以加强MLLM对图表视觉布局和数据细节的理解。为了促进空间和文本学习，我们提出了START数据集，该数据集通过一种新的数据生成管道生成，首先利用MLLM将真实图表图像转换为可执行的图表代码，恢复底层数据表示的同时保持真实世界图表的视觉分布。然后通过大型语言模型（LLM）进化代码，以确定捕捉图表视觉结构的图表元素位置，解决现有方法无法处理的挑战。为了评估模型对图表空间结构的理解能力，我们提出了图表空间理解基准（CS-Bench），填补了全面图表理解评估中的关键空白。借助空间和文本学习，START在模型大小和基准方面均实现了持续提升，超过了基础模型并在先前的最先进水平上取得了明显优势。代码、数据和模型将向公众开放。
英文摘要：
Chart understanding is crucial for deploying multimodal large language models (MLLMs) in real-world scenarios such as analyzing scientific papers and technical reports. Unlike natural images, charts pair a structured visual layout (spatial property) with an underlying data representation (textual property) -- grasping both is essential for precise, fine-grained chart reasoning. Motivated by this observation, we propose START, the Spatial and Textual learning for chART understanding. Specifically, we introduce (i) chart-element grounding and (ii) chart-to-code generation to strengthen an MLLM's understanding of both chart visual layout and data details. To facilitate spatial and textual learning, we propose the START-Dataset generated with a novel data-generation pipeline that first leverages an MLLM to translate real chart images into executable chart code, recovering the underlying data representation while preserving the visual distribution of real-world charts. We then evolve the code with a Large Language Model (LLM) to ascertain the positions of chart elements that capture the chart's visual structure, addressing challenges that existing methods cannot handle. To evaluate a model's ability to understand chart spatial structures, we propose the Chart Spatial understanding Benchmark (CS-Bench), filling a critical gap in comprehensive chart understanding evaluation. Leveraging spatial and textual learning, START delivers consistent gains across model sizes and benchmarks over the base models and surpasses prior state-of-the-art by a clear margin. Code, data and models will be publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
