With the development of web technology, multi-modal or multi-view data has surged as a major stream for big data, where each modal/view encodes individual property of data objects. Often, different modalities are complementary to each other. Such fact motivated a lot of research attention on fusing the multi-modal feature spaces to comprehensively characterize the data objects. Most of the existing state-of-the-art focused on how to fuse the energy or information from multi-modal spaces to deliver a superior performance over their counterparts with single modal. Recently, deep neural networks have exhibited as a powerful architecture to well capture the nonlinear distribution of high-dimensional multimedia data, so naturally does for multi-modal data. Substantial empirical studies are carried out to demonstrate its advantages that are benefited from deep multi-modal methods, which can essentially deepen the fusion from multi-modal deep feature spaces. In this paper, we provide a substantial overview of the existing state-of-the-arts on the filed of multi-modal data analytics from shallow to deep spaces. Throughout this survey, we further indicate that the critical components for this field go to collaboration, adversarial competition and fusion over multi-modal spaces. Finally, we share our viewpoints regarding some future directions on this field.
Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration
Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection
Xidian University（西安电子科技大学）；Nanyang Technological University（南洋理工大学）；Zhengzhou University（郑州大学）；China University of Geosciences（中国地质大学）；Swinburne University of Technology（斯威本科技大学）
Event-based Video Person Re-identification via Cross-Modality and Temporal Collaboration
School of Computer Science and Technology, Wuhan University of Science and Technology（武汉科技大学计算机科学与技术学院）；Hubei Province Key Laboratory of Intelligent Information Processing and Real-Time Industrial System（湖北省智能信息处理与实时工业系统重点实验室）
Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning
Institute of Education, National Cheng Kung University（国立成功大学教育研究所）；Institute of Intelligent System, National Yang Ming Chiao Tung University（阳明交通大学智能系统研究所）；Department of Computer Science, University at Albany, State University of New York（纽约州立大学水牛城分校计算机科学系）
A Distributed Multi-Modal Sensing Approach for Human Activity Recognition in Real-Time Human-Robot Collaboration
University of Genoa（热那亚大学）；The University of Danang–University of Science and Technology（丹江-科学技术大学）；Japan Advanced Institute of Science and Technology（日本先进科学研究院）
Multimodal Collaboration Networks for Geospatial Vehicle Detection in Dense, Occluded, and Large-Scale Events
Beijing University of Posts and Telecommunications（北京邮电大学）；Fuzhou University（福州大学）；Univ. Grenoble Alpes（格勒诺布尔大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔综合理工学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；German Aerospace Center（德国航空航天中心）
Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
Department of Computer Science, University of Kaiserslautern-Landau (RPTU)（科斯拉尔特伦大学计算机科学系）；SDS, German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；INRAE, UMR TETIS, University of Montpellier（蒙彼利埃大学UMR TETIS）；CIRAD, UMR TETIS, University of Montpellier（蒙彼利埃大学UMR TETIS）；INRIA, EVERGREEN, University of Montpellier（蒙彼利埃大学）
AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理学院计算与信息系统学院）；School of Artifical Intelligence, Xidian University（西安电子科技大学人工智能学院）
MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（计算机科学与工程系，香港科学与技术大学）
Dissecting RGB-D Learning for Improved Multi-modal Fusion
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及跨学科应用国家重点实验室（东南大学））；Tsinghua University（清华大学）；College of Computer Science, Beijing University of Technology（北京理工大学计算机学院）
Hollywood Town: Long-Video Generation via Cross-Modal Multi-Agent Orchestration
The Hong Kong University of Science and Technology（香港科学与技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；University of Chicago（芝加哥大学）；Stanford University（斯坦福大学）
Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition
Great Bay University（大湾大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Tencent Youtu Lab（腾讯优图实验室）
Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）
REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge
University of Leicester（莱斯特大学）；University of Cambridge（剑桥大学）；Shenzhen University（深圳大学）；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；University of Nottingham（诺丁汉大学）；University of Augsburg（奥格斯堡大学）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
Nanjing University of Science and Technology（南京理工大学）；Baidu Inc（百度公司）；AIML, Adelaide University（阿德莱德大学AIML）；SUTD（新加坡科技设计大学）；Southeast University（东南大学）；East China Normal University（华东师范大学）；University of Oxford（牛津大学）
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
GFZ Helmholtz Center for Geosciences（德国波茨坦地学研究中心亥姆霍兹中心）；INRAE, UMR TETIS, University of Montpellier（法国蒙彼利埃大学农业环境与生态研究院、蒙彼利埃大学TETIS联合研究单位）；INRIA, EVERGREEN, University of Montpellier（法国蒙彼利埃大学信息与自动化研究所、EVERGREEN实验室）；CIRAD, UMR TETIS, University of Montpellier（法国蒙彼利埃大学国际农业研究磋商组织、蒙彼利埃大学TETIS联合研究单位）
VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows
Anhui University（安徽大学）；Peng Cheng Laboratory（鹏城实验室）；Peking University（北京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Sydney（悉尼大学）；La Trobe University（乐卓博大学）；University of Science and Technology of China（中国科学技术大学）
Residual Cross-Modal Fusion Networks for Audio-Visual Navigation
School of Computer Science and Technology, Xinjiang University, Urumqi, China（新疆大学计算机科学与技术学院）；Joint International Research Laboratory of Silk Road Multilingual Cognitive（丝绸之路多语认知联合国际实验室）；School of Mechatronic Engineering and Automation Shanghai University, Shanghai, China（上海大学机电工程与自动化学院）
