Due to the rapid advancements of sensory and computing technology, multi-modal data sources that represent the same pattern or phenomenon have attracted growing attention. As a result, finding means to explore useful information from these multi-modal data sources has quickly become a necessity. In this paper, a discriminative vectorial framework is proposed for multi-modal feature representation in knowledge discovery by employing multi-modal hashing (MH) and discriminative correlation maximization (DCM) analysis. Specifically, the proposed framework is capable of minimizing the semantic similarity among different modalities by MH and exacting intrinsic discriminative representations across multiple data sources by DCM analysis jointly, enabling a novel vectorial framework of multi-modal feature representation. Moreover, the proposed feature representation strategy is analyzed and further optimized based on canonical and non-canonical cases, respectively. Consequently, the generated feature representation leads to effective utilization of the input data sources of high quality, producing improved, sometimes quite impressive, results in various applications. The effectiveness and generality of the proposed framework are demonstrated by utilizing classical features and deep neural network (DNN) based features with applications to image and multimedia analysis and recognition tasks, including data visualization, face recognition, object recognition; cross-modal (text-image) recognition and audio emotion recognition. Experimental results show that the proposed solutions are superior to state-of-the-art statistical machine learning (SML) and DNN algorithms.
Interventional Imbalanced Multi-Modal Representation Learning via $β$-Generalization Front-Door Criterion
Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Thrust of Artificial Intelligence, the Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)人工智能学域);the Hong Kong University of Science and Technology(香港科技大学)
Credibility-Aware Multi-Modal Fusion Using Probabilistic Circuits
Erik Jonsson School of Engineering & Computer Science, The University of Texas at Dallas(德克萨斯大学达拉斯分校埃里克·琼森工程与计算机科学学院);Air Force Research Laboratory(空军研究实验室);TU Darmstadt(达姆施塔特工业大学)
MDENet: Multi-modal Dual-embedding Networks for Malware Open-set Recognition
The Hong Kong Polytechnic University(香港理工大学);The Hong Kong Polytechnic University Shenzhen Research Institute(香港理工大学深圳研究院);University of Electronic Science and Technology of China(电子科技大学);Beijing Institute of Technology(北京理工大学);Jinan University(暨南大学)
Interpretable Generative and Discriminative Learning for Multimodal and Incomplete Clinical Data
Department of Health Sciences and Technology (DHEST), ETH Zurich(健康科学与技术系,苏黎世联邦理工学院);Department of Signal Theory and Communications, Universidad Carlos III de Madrid(信号理论与通信系,卡洛斯三世大学);UCL Hawkes Institute, Department of Computer Science, University College London(UCL Hawkes研究所,伦敦大学学院计算机科学系);Instituto de Investigación Sanitaria Gregorio Marañón (IiSGM), Madrid(Gregorio Marañón卫生研究所(IiSGM),马德里)
Hierarchical Multi-to-Single-Modal Knowledge Distillation for Disruption Prediction in EAST
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);Institute of Plasma Physics, Chinese Academy of Sciences(中国科学院等离子体物理研究所);State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University(安徽大学光电子信息获取与控制技术国家重点实验室)
GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning
Carnegie Mellon University(卡内基梅隆大学)
3D多目标跟踪(MOT)是自主系统的关键。最近的工作使用标准的检测跟踪流水线,其中首先对每个对象独立地执行特征提取,以便计算亲和矩阵。然后将亲和矩阵传递给匈牙利算法进行数据关联。这个标准管道的一个关键过程是学习不同对象的区别特征,以减少数据关联过程中的混淆。在这项工作中,我们提出了两种技术,以提高区分特征学习MOT:(1)而不是获得每个对象的特征独立,我们提出了一种新的特征交互机制,通过引入图神经网络。结果,一个对象的特征被告知其他对象的特征,使得对象特征可以倾向于具有相似特征的对象(即,可能具有相同ID的对象)并且偏离具有不同特征的对象(即,对象可能具有不同的ID),从而为每个对象提供更具鉴别力的特征;(2)在先前的工作中,我们提出了一种新的联合特征提取器,以同时从2D和3D空间学习外观和运动特征,而不是从2D或3D空间获得特征。由于来自不同模态的特征通常具有互补信息,因此联合特征可以比来自每个单独模态的特征更具区分性。为了确保联合特征提取器不严重依赖于一种模态,我们还提出了一种集成训练范式。通过广泛的评估,我们提出的方法在KITTI和nuScenes 3D MOT基准测试中达到了最先进的性能。我们的代码将在https://github.com/xinshuoweng/GNN3DMOT上提供