Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Baidu Inc（百度公司）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Although great progress has been made by previous table understanding methods including recent approaches based on large language models (LLMs), they rely heavily on the premise that given tables must be converted into a certain text sequence (such as Markdown or HTML) to serve as model input. However, it is difficult to access such high-quality textual table representations in some real-world scenarios, and table images are much more accessible. Therefore, how to directly understand tables using intuitive visual information is a crucial and urgent challenge for developing more practical applications. In this paper, we propose a new problem, multimodal table understanding, where the model needs to generate correct responses to various table-related requests based on the given table image. To facilitate both the model training and evaluation, we construct a large-scale dataset named MMTab, which covers a wide spectrum of table images, instructions and tasks. On this basis, we develop Table-LLaVA, a generalist tabular multimodal large language model (MLLM), which significantly outperforms recent open-source MLLM baselines on 23 benchmarks under held-in and held-out settings. The code and data is available at this https://github.com/SpursGoZmy/Table-LLaVA
Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables
State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；University of Science and Technology of China（中国科学技术大学）；Artificial Intelligence Research Institute（人工智能研究院）；iFLYTEK Co., Ltd（iFLYTEK公司）
Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)（德国人工智能研究中心有限公司）；Technische Universität Berlin（柏林技术大学）；BIFOLD；Deutsche Telekom（德国电信）；Common Crawl Foundation（Common Crawl基金会）；Humboldt-Universität zu Berlin（柏林洪堡大学）
TABQAWORLD: Optimizing Multimodal Reasoning for Multi-Turn Table Question Answering
University of California, Los Angeles（加州大学洛杉矶分校）；McGill University（麦吉尔大学）；Université de Montréal（蒙特利尔大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；SimpleWay
MultiModalQA: Complex Question Answering over Text, Tables and Images
The Allen Institute for AI（艾伦人工智能研究所）；Tel-Aviv University（特拉维夫大学）；University of Washington（华盛顿大学）
在回答复杂的问题时，人们可以无缝地结合来自视觉，文本和表格来源的信息。虽然近年来对多个证据推理的模型的兴趣激增，但对跨多个模态推理的问答模型的研究相对较少。在本文中，我们提出了MultiModalQA（MMQA）：一个具有挑战性的问答数据集，需要联合推理的文本，表格和图像。我们使用一个新的框架来创建MMQA，该框架用于大规模生成复杂的多模态问题，从维基百科获取表格，并使用每个表格中出现的实体附加图像和文本段落。然后，我们定义了一个形式语言，使我们能够采取的问题，可以回答从一个单一的模态，并将它们结合起来，以产生跨模态的问题。最后，众包工作人员将这些自动生成的问题转化为更流畅的语言。我们通过这个过程创建了29，918个问题，并以经验证明了多模态多跳方法解决我们任务的必要性：我们的多跳模型ImplicitDecomp在跨模态问题上的平均F1为51.7，大大超过了达到38.2 F1的强大基线，但仍然明显落后于人类表现，这是90.1 F1
Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Research Center for Scientific Data Hub, Zhejiang Lab, Hangzhou, China（浙江实验室科学数据中心研究中心）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）
Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach
University of Science and Technology of China（中国科学技术大学）；Baidu Research（百度研究院）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangzhou HKUST Fok Ying Tung Research Institute（广州香港科技大学霍英东研究院）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
TabularMath: Understanding Math Reasoning over Tables with Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
TableLoRA: Low-rank Adaptation on Table Structure Understanding for Large Language Models
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；Peking University（北京大学）；Microsoft（微软公司）
Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；Singapore University of Technology and Design（新加坡科技设计大学）；Shanghai Jiao Tong University（上海交通大学）
TableTime: Reformulating Time Series Classification as Training-Free Table Understanding with Large Language Models
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；Artificial Intelligence Research Institute, iFLYTEK Co., Ltd（人工智能研究院，iFLYTEK公司）
HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）；Huawei Technologies Co., Ltd（华为技术有限公司）
Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University, China（新型软件技术国家重点实验室，南京大学，中国）；School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学，中国）；School of Intelligence Science and Technology, Nanjing University, China（智能科学与技术学院，南京大学，中国）
