arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01326cs.CV
收藏

TabPedia:通过概念协同实现全面的视觉表格理解

TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy

University of Science and Technology of China(中国科学技术大学);ByteDance Inc.(字节跳动公司)

Weichao Zhao, Hao Feng, Qi Liu, Jingqun Tang, Shu Wei, Binghong Wu, Lei Liao, Yongjie Ye, Hao Liu, Wengang Zhou, Houqiang Li, Can Huang

AI总结:

本文提出新型视觉语言模型 TabPedia,通过概念协同机制统一集成表格检测、识别、查询和问答等视觉表格理解任务,并构建含约9000个QA对的基准 ComTQA,验证了利用LLMs实现全面视觉表格理解的有效性。

摘要:

表格包含事实和定量数据,并伴随各种结构和内容,这给机器理解带来了挑战。以往的方法通常为单个任务设计特定任务的架构和目标,导致模态隔离和复杂的工作流程。在本文中,我们提出了一种新型大型视觉语言模型 TabPedia,配备了概念协同机制。在该机制中,所有涉及的多样化视觉表格理解(VTU)任务和多源视觉嵌入都被抽象为概念。这一统一框架允许 TabPedia 通过利用大型语言模型(LLMs)的能力,无缝集成 VTU 任务,如表格检测、表格结构识别、表格查询和表格问答。此外,概念协同机制使表格感知相关和理解决相关的任务能够协同工作,因为它们可以有效地从相应的源感知嵌入中利用所需的线索。此外,为了更好地评估真实场景中的 VTU 任务,我们建立了一个新的综合表格 VQA 基准 ComTQA,包含约 9,000 个 QA 对。在多个公共基准上进行的广泛定量和定性实验,涵盖表格感知和理解决任务,验证了我们的 TabPedia 的有效性。卓越的性能进一步证实了当所有概念协同工作时,使用 LLMs 理解视觉表格的可行性。基准 ComTQA 已在 https://huggingface.co/datasets/ByteDance/ComTQA 开源。源代码和模型也已发布在 https://github.com/zhaowc-ustc/TabPedia。

英文摘要:

Tables contain factual and quantitative data accompanied by various structures and contents that pose challenges for machine comprehension. Previous methods generally design task-specific architectures and objectives for individual tasks, resulting in modal isolation and intricate workflows. In this paper, we present a novel large vision-language model, TabPedia, equipped with a concept synergy mechanism. In this mechanism, all the involved diverse visual table understanding (VTU) tasks and multi-source visual embeddings are abstracted as concepts. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering, by leveraging the capabilities of large language models (LLMs). Moreover, the concept synergy mechanism enables table perception-related and comprehension-related tasks to work in harmony, as they can effectively leverage the needed clues from the corresponding source perception embeddings. Furthermore, to better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring approximately 9,000 QA pairs. Extensive quantitative and qualitative experiments on both table perception and comprehension tasks, conducted across various public benchmarks, validate the effectiveness of our TabPedia. The superior performance further confirms the feasibility of using LLMs for understanding visual tables when all concepts work in synergy. The benchmark ComTQA has been open-sourced at https://huggingface.co/datasets/ByteDance/ComTQA. The source code and model also have been released athttps://github.com/zhaowc-ustc/TabPedia.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑