Bridge the Gap between Language models and Tabular Understanding
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Microsoft（微软）；Duke University（杜克大学）；Yale University（耶鲁大学）；Peking University（北京大学）
Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Chenyu You, Jianhui Chang, Daxin Jiang, Jia Li
Table pretrain-then-finetune paradigm has been proposed and employed at a rapid pace after the success of pre-training in the natural language domain. Despite the promising findings in tabular pre-trained language models (TPLMs), there is an input gap between pre-training and fine-tuning phases. For instance, TPLMs jointly pre-trained with table and text input could be effective for tasks also with table-text joint input like table question answering, but it may fail for tasks with only tables or text as input such as table retrieval. To this end, we propose UTP, an approach that dynamically supports three types of multi-modal inputs: table-text, table, and text. Specifically, UTP is pre-trained with two strategies: (1) We first utilize a universal mask language modeling objective on each kind of input, enforcing the model to adapt various inputs. (2) We then present Cross-Modal Contrastive Regularization (CMCR), which utilizes contrastive learning to encourage the consistency between table-text cross-modality representations via unsupervised instance-wise training signals during pre-training. By these means, the resulting model not only bridges the input gap between pre-training and fine-tuning but also advances in the alignment of table and text. Extensive results show UTP achieves superior results on uni-modal input tasks (e.g., table retrieval) and cross-modal input tasks (e.g., table question answering).
Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution
Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学陈笃生公共卫生学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究院）
NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data
Department of Information Systems, Faculty of Computing and Information Technology（信息系统系，计算与信息科技学院）；King Abdulaziz University（国王阿卜杜勒阿齐兹大学）；Department of Information Systems and Technology, College of Computer Science and Engineering（信息系统与技术系，计算机科学与工程学院）；University of Jeddah（朱德赫大学）；School of Computing and Creative Technologies（计算与创意科技学院）；University of the West of England（西英格兰大学）
Small Models are LLM Knowledge Triggers on Medical Tabular Prediction
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Thrust of Artificial Intelligence, Information Hub, HKUST (GZ)（香港科技大学（广州）信息枢纽人工智能方向）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Second Affiliated Hospital Zhejiang University School of Medicine（浙江大学医学院附属第二医院）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）
RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Institute of Artificial Intelligence and Future Networks, Beijing Normal University（北京师范大学人工智能与未来网络研究院）；Elmleaf Ltd.（榆叶有限公司）；BNU-UIC Institute of Artificial Intelligence and Future Networks, Beijing Normal University (Zhuhai)（北京师范大学-香港浸会大学联合国际学院人工智能与未来网络研究院）；Guangdong Key Lab of AI and Multi-Modal Data Processing, BNU-HKBU United International College（广东省人工智能与多模态数据处理重点实验室，北京师范大学-香港浸会大学联合国际学院）
MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
Technion – Israel Institute of Technology（技术ion – 以色列理工学院）；Prior Labs（Prior实验室）；NVIDIA；SODA Team, INRIA Saclay, Palaiseau（SODA团队，INRIA萨克莱，帕莱索）；University of Freiburg（弗赖堡大学）；Probabl；ELLIS Institute Tübingen（图宾根ELLIS研究所）
