In this paper, we propose the Discriminative Multiple Canonical Correlation Analysis (DMCCA) for multimodal information analysis and fusion. DMCCA is capable of extracting more discriminative characteristics from multimodal information representations. Specifically, it finds the projected directions which simultaneously maximize the within-class correlation and minimize the between-class correlation, leading to better utilization of the multimodal information. In the process, we analytically demonstrate that the optimally projected dimension by DMCCA can be quite accurately predicted, leading to both superior performance and substantial reduction in computational cost. We further verify that Canonical Correlation Analysis (CCA), Multiple Canonical Correlation Analysis (MCCA) and Discriminative Canonical Correlation Analysis (DCCA) are special cases of DMCCA, thus establishing a unified framework for Canonical Correlation Analysis. We implement a prototype of DMCCA to demonstrate its performance in handwritten digit recognition and human emotion recognition. Extensive experiments show that DMCCA outperforms the traditional methods of serial fusion, CCA, MCCA and DCCA.
Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement
University of Wolverhampton(伍尔弗汉普顿大学);São Paulo State University(圣保罗州立大学);TECNALIA(泰克纳利亚研究中心);Basque Research & Technology Alliance(巴斯克研究与技术联盟);University of the Basque Country(巴斯克大学);Edinburgh Napier University(爱丁堡龙比亚大学)
Robust Multi-View Learning via Representation Fusion of Sample-Level Attention and Alignment of Simulated Perturbation
University of Electronic Science and Technology of China(电子科技大学);Hainan University(海南大学);Singapore University of Technology and Design(新加坡科技设计大学);Southeast University(东南大学);The University of Tokyo(东京大学)
Local Sample-weighted Multiple Kernel Clustering with Consensus Discriminative Graph
National University of Defense Technology(国防科技大学);Hunan University(湖南大学);Supercomputing and Cloud Computing Institute(超级计算与云计算研究院);State University of New York, New Paltz(纽约州立大学新帕尔茨分校)
Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement
College of Computer Science, Sichuan University(四川大学计算机科学学院);Engineering Research Center of Machine Learning(机器学习工程研究中心);China Telecom Institute of AI(中国电信人工智能研究院)