用于多模态学习的神经启发信息论分层感知
Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning
AI总结:
受神经科学启发提出基于信息瓶颈的ITHP模型,通过指定主模态并平衡互信息构建紧凑信息流,在多模态学习任务中显著提升性能并在CMU-MOSI数据集上超越人类水平。
摘要:
在自主系统和信息物理系统中,整合与处理来自各种来源或模态的信息对于获得对现实世界全面而准确的感知至关重要。受神经科学启发,我们开发了信息论分层感知(ITHP)模型,该模型利用了信息瓶颈的概念。与大多数在神经网络中同等纳入所有模态的传统融合模型不同,我们的模型指定一个主模态,并将其余模态视为信息通路中的检测器,用于提炼信息流。我们提出的感知模型侧重于构建有效且紧凑的信息流,方法是在最小化潜在状态与输入模态状态之间的互信息,以及最大化潜在状态与其余模态状态之间的互信息之间取得平衡。这种方法产生了紧凑的潜在状态表示,既保留了相关信息又最小化了冗余,从而大幅提升了多模态表示学习的性能。在MUStARD、CMU-MOSI和CMU-MOSEI数据集上的实验评估表明,我们的模型在多模态学习场景中持续提炼关键信息,优于最先进的基准。值得注意的是,在CMU-MOSI数据集上,ITHP在多模态情感二分类任务的所有评估指标(即二分类准确率、F1分数、平均绝对误差和皮尔逊相关系数)上均超越了人类水平的表现。
英文摘要:
Integrating and processing information from various sources or modalities are critical for obtaining a comprehensive and accurate perception of the real world in autonomous systems and cyber-physical systems. Drawing inspiration from neuroscience, we develop the Information-Theoretic Hierarchical Perception (ITHP) model, which utilizes the concept of information bottleneck. Different from most traditional fusion models that incorporate all modalities identically in neural networks, our model designates a prime modality and regards the remaining modalities as detectors in the information pathway, serving to distill the flow of information. Our proposed perception model focuses on constructing an effective and compact information flow by achieving a balance between the minimization of mutual information between the latent state and the input modal state, and the maximization of mutual information between the latent states and the remaining modal states. This approach leads to compact latent state representations that retain relevant information while minimizing redundancy, thereby substantially enhancing the performance of multimodal representation learning. Experimental evaluations on the MUStARD, CMU-MOSI, and CMU-MOSEI datasets demonstrate that our model consistently distills crucial information in multimodal learning scenarios, outperforming state-of-the-art benchmarks. Remarkably, on the CMU-MOSI dataset, ITHP surpasses human-level performance in the multimodal sentiment binary classification task across all evaluation metrics (i.e., Binary Accuracy, F1 Score, Mean Absolute Error, and Pearson Correlation).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。