TIP：面向不完整数据多模态分类的表格-图像预训练
TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data
AI总结：
针对表格数据异质缺失、与图像模态差异大且现有方法未考虑不完整场景的问题，提出含掩码重建、匹配对比学习及专用编码器与交互模块的TIP预训练框架，在多类数据集的完整/不完整场景分类任务中性能超越SOTA。
摘要：
图像与结构化表格是现实世界数据库的核心组成部分。尽管表格-图像表示学习有望催生新的研究洞见，但该任务仍颇具挑战——表格数据通常具有异质性且存在缺失，与图像之间存在显著的模态差异。现有研究多聚焦于完整数据场景下的简单模态融合策略，未考虑数据缺失问题，因此实际应用存在局限。本文提出TIP，一种新型表格-图像预训练框架，用于学习对不完整表格数据具有鲁棒性的多模态表示。具体而言，TIP探索了一种全新的自监督学习（SSL）策略：包含用于解决数据缺失问题的掩码表格重建任务，以及用于捕获多模态信息的图像-表格匹配和对比学习目标。此外，TIP还设计了适配不完整、异质表格数据的通用表格编码器，以及用于跨模态表示学习的多模态交互模块。我们在自然图像和医学图像数据集上开展了下游多模态分类任务实验。结果表明，在完整与不完整数据场景下，TIP的表现均优于当前最优的有监督/自监督学习图像/多模态算法。我们的代码可在https://github.com/siyi-wind/TIP获取。
英文摘要：
Images and structured tables are essential parts of real-world databases. Though tabular-image representation learning is promising to create new insights, it remains a challenging task, as tabular data is typically heterogeneous and incomplete, presenting significant modality disparities with images. Earlier works have mainly focused on simple modality fusion strategies in complete data scenarios, without considering the missing data issue, and thus are limited in practice. In this paper, we propose TIP, a novel tabular-image pre-training framework for learning multimodal representations robust to incomplete tabular data. Specifically, TIP investigates a novel self-supervised learning (SSL) strategy, including a masked tabular reconstruction task for tackling data missingness, and image-tabular matching and contrastive learning objectives to capture multimodal information. Moreover, TIP proposes a versatile tabular encoder tailored for incomplete, heterogeneous tabular data and a multimodal interaction module for inter-modality representation learning. Experiments are performed on downstream multimodal classification tasks using both natural and medical image datasets. The results show that TIP outperforms state-of-the-art supervised/SSL image/multimodal algorithms in both complete and incomplete data scenarios. Our code is available at https://github.com/siyi-wind/TIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
