arXiv 2506.040882025-06-04cs.LGcs.AIcs.CLcs.CV

利用特权结构化信息的多模态表格推理

Multimodal Tabular Reasoning with Privileged Structured Information

School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学）；National Key Laboratory for Novel Software Technology, Nanjing University, China（新型软件技术国家重点实验室，南京大学）；AI Business, Alibaba Group（阿里集团人工智能业务）

Jun-Peng Jiang, Yu Xia, Hai-Long Sun, Shiyin Lu, Qing-Guo Chen, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, Han-Jia Ye

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Turbo框架，利用训练时的特权结构化表格信息，通过基于DeepSeek-R1的结构感知推理轨迹生成和有利推理路径筛选，在仅9k数据下实现多模态表格推理性能提升7.2%。

摘要：

表格推理涉及对表格数据进行多步信息提取和逻辑推断。尽管近期进展利用大语言模型（LLMs）对结构化表格进行推理，但在现实场景中，这种高质量的文本表示往往不可得，表格通常以图像形式出现。本文针对表格图像中的表格推理任务，利用训练期间可用的特权结构化信息来增强多模态大语言模型（MLLMs）。关键挑战在于准确对齐结构化信息与视觉表示的复杂性，以及在输入模态存在差异的情况下有效将结构化推理能力迁移到多模态大语言模型。为解决这些问题，我们提出了TabUlar Reasoning with Bridged infOrmation（Turbo），这是一个利用特权结构化表格进行多模态表格推理的新框架。Turbo受益于基于DeepSeek-R1的结构感知推理轨迹生成器，有助于生成高质量的模态桥接数据。在此基础上，Turbo反复生成并选择有利的推理路径，进一步增强模型的表格推理能力。实验结果表明，在仅有9k数据的有限条件下，Turbo在多个数据集上取得了最先进的性能（相比此前SOTA提升+7.2%）。

英文摘要：

Tabular reasoning involves multi-step information extraction and logical inference over tabular data. While recent advances have leveraged large language models (LLMs) for reasoning over structured tables, such high-quality textual representations are often unavailable in real-world settings, where tables typically appear as images. In this paper, we tackle the task of tabular reasoning from table images, leveraging privileged structured information available during training to enhance multimodal large language models (MLLMs). The key challenges lie in the complexity of accurately aligning structured information with visual representations, and in effectively transferring structured reasoning skills to MLLMs despite the input modality gap. To address these, we introduce TabUlar Reasoning with Bridged infOrmation ({\sc Turbo}), a new framework for multimodal tabular reasoning with privileged structured tables. {\sc Turbo} benefits from a structure-aware reasoning trace generator based on DeepSeek-R1, contributing to high-quality modality-bridged data. On this basis, {\sc Turbo} repeatedly generates and selects the advantageous reasoning paths, further enhancing the model's tabular reasoning ability. Experimental results demonstrate that, with limited ($9$k) data, {\sc Turbo} achieves state-of-the-art performance ($+7.2\%$ vs. previous SOTA) across multiple datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。