arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1904.01775cs.LGeess.SP
收藏

基于深度多集典型相关的多模态表示学习

Multimodal Representation Learning using Deep Multiset Canonical Correlation

University of Southern California(南加州大学)

Krishna Somandepalli, Naveen Kumar, Ruchir Travadi, Shrikanth Narayanan

摘要:

我们提出了深度多集典型相关分析(dMCCA)作为使用CCA的表示学习的扩展,当底层信号在多个(两个以上)模态中观察时。我们使用深度学习框架来学习从不同模态到共享子空间的非线性变换,使得表示最大化观测的模态间和模态内协方差的比率。与线性判别分析不同,我们不需要类信息来学习这些表示,并且我们证明了这个模型可以使用小批量来训练复杂数据。使用合成数据实验,我们表明,dMCCA可以有效地恢复共同的信号跨越不同的模态破坏的乘性和加性噪声。我们还分析了我们的模型的敏感性,以恢复相关的组件相对于小批量的大小和嵌入的尺寸。在有噪声的手写数据集上的性能评估表明,我们的模型优于其他基于CCA的方法,并且与在该数据集上进行端到端训练的深度神经网络模型相当。

英文摘要:

We propose Deep Multiset Canonical Correlation Analysis (dMCCA) as an extension to representation learning using CCA when the underlying signal is observed across multiple (more than two) modalities. We use deep learning framework to learn non-linear transformations from different modalities to a shared subspace such that the representations maximize the ratio of between- and within-modality covariance of the observations. Unlike linear discriminant analysis, we do not need class information to learn these representations, and we show that this model can be trained for complex data using mini-batches. Using synthetic data experiments, we show that dMCCA can effectively recover the common signal across the different modalities corrupted by multiplicative and additive noise. We also analyze the sensitivity of our model to recover the correlated components with respect to mini-batch size and dimension of the embeddings. Performance evaluation on noisy handwritten datasets shows that our model outperforms other CCA-based approaches and is comparable to deep neural network models trained end-to-end on this dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑