Pre-trained language models are still far from human performance in tasks that need understanding of properties (e.g. appearance, measurable quantity) and affordances of everyday objects in the real world since the text lacks such information due to reporting bias. In this work, we study whether integrating visual knowledge into a language model can fill the gap. We investigate two types of knowledge transfer: (1) text knowledge transfer using image captions that may contain enriched visual knowledge and (2) cross-modal knowledge transfer using both images and captions with vision-language training objectives. On 5 downstream tasks that may need visual knowledge to solve the problem, we perform extensive empirical comparisons over the presented objectives. Our experiments show that visual knowledge transfer can improve performance in both low-resource and fully supervised settings.
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件学院）
3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Huazhong University of Science and Technology（华中科技大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）
Leveraging Graph-based Cross-modal Information Fusion for Neural Sign Language Translation
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）；City University of Hong Kong（香港城市大学）；Xiamen University（厦门大学）
CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
Instituto de Computação, Universidade Estadual de Campinas (UNICAMP), Brasil（计算机学院，Campinas州立大学（UNICAMP））；Instituto de Estudos da Linguagem, Universidade Estadual de Campinas (UNICAMP), Brasil（语言研究学院，Campinas州立大学（UNICAMP））；Instituto de Informática, Universidade Federal de Goiás (UFG), Goiás, Brasil（信息学院，戈亚斯联邦大学（UFG））；Department of Computer Science, University of Sheffield, Sheffield, United Kingdom（计算机科学系，谢菲尔德大学）
MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction
Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）；School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）；Information Technology Center, Nagoya University（名古屋大学信息技术中心）
Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Zhejiang University（浙江大学）；Shanghai Innovation Institute（上海创新研究院）；Huazhong University of Science and Technology（华中科技大学）；University of Science and Technology of China（中国科学技术大学）；Zhongguancun Academy（中关村学院）；Shanghai AI Laboratory（上海人工智能实验室）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
