ChartPoint: 通过 grounding 反射引导 MLLMs 进行图表推理
ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning
AI总结：
ChartPoint通过引入 grounding 反射机制，提升 MLLMs 在图表推理中的表现，开发了两个指令微调模型，在多个图表基准上取得显著提升。
摘要：
多模态大语言模型（MLLMs）已逐渐成为图表理解的强大工具。然而，它们严重依赖OCR提取内容，当图表文本注释稀疏时会导致数值幻觉。尽管现有方法侧重于扩展指令，但未能解决根本挑战，即视觉感知推理。在本文中，我们提出一个关键观察：MLLMs在图表元素和比例关系上存在弱 grounding，这体现在它们无法定位关键位置以匹配推理。为弥合这一差距，我们提出了PointCoT，它在图表中将反思交互整合到推理链中。通过提示MLLMs生成边界框并基于位置注释重新绘制图表，我们建立了文本推理步骤与视觉 grounding 区域之间的联系。我们进一步引入自动化流程构建ChartPoint-SFT-62k数据集，该数据集包含19.2K高质量图表样本，具有逐步CoT、边界框和重新绘制的可视化。利用此数据，我们开发了两个指令微调模型，ChartPointQ2和ChartPointQ2.5，在多个图表基准上优于最先进模型，例如在ChartBench上提升了+5.04%。
英文摘要：
Multimodal Large Language Models (MLLMs) have emerged as powerful tools for chart comprehension. However, they heavily rely on extracted content via OCR, which leads to numerical hallucinations when chart textual annotations are sparse. While existing methods focus on scaling instructions, they fail to address the fundamental challenge, i.e., reasoning with visual perception. In this paper, we identify a critical observation: MLLMs exhibit weak grounding in chart elements and proportional relationships, as evidenced by their inability to localize key positions to match their reasoning. To bridge this gap, we propose PointCoT, which integrates reflective interaction into chain-of-thought reasoning in charts. By prompting MLLMs to generate bounding boxes and re-render charts based on location annotations, we establish connections between textual reasoning steps and visual grounding regions. We further introduce an automated pipeline to construct ChartPoint-SFT-62k, a dataset featuring 19.2K high-quality chart samples with step-by-step CoT, bounding box, and re-rendered visualizations. Leveraging this data, we develop two instruction-tuned models, ChartPointQ2 and ChartPointQ2.5, which outperform state-of-the-art across several chart benchmarks, e.g., +5.04\% on ChartBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
