arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.13782cs.LGcs.AI
收藏

Multimodal Co-learning:挑战、数据集应用、最新进展和未来方向

Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions

Symbiosis Institute of Technology (SIT), Symbiosis International (Deemed University)(共生国际(视为大学)共生技术学院);Symbiosis Centre for Applied Artificial Intelligence, Symbiosis Institute of Technology (SIT), Symbiosis International (Deemed University)(共生国际(视为大学)共生技术学院共生应用人工智能中心);University of Winnipeg(温尼伯大学)

Anil Rahate, Rahee Walambe, Sheela Ramanna, Ketan Kotecha

摘要:

多模态深度学习系统采用多种模态,如文本、图像、音频、视频等,与单个模式相比,表现出更好的性能(即,单峰)系统。多模态机器学习涉及多个方面:表示、翻译、对齐、融合和协同学习。在多模态机器学习的当前状态下,假设所有模态在训练和测试期间都存在、对齐并且无噪声。然而,在现实世界的任务中,通常,观察到一个或多个模态丢失、有噪声、缺乏注释数据、具有不可靠的标签,并且在训练或测试和/或两者中是稀缺的。这一挑战由一种称为多模式共同学习的学习范式来解决。(资源贫乏)模态的建模通过利用来自另一个(资源丰富)模态的知识来辅助,该知识使用模态之间的知识转移,包括它们的表示和预测模型。共同学习是一个新兴领域,没有专门的审查明确侧重于共同学习解决的所有挑战。为此,在这项工作中,我们对尚未完全探索的多模式共同学习的新兴领域进行了全面调查。我们回顾了克服一个或多个共同学习挑战的实现,而没有明确地将它们视为共同学习挑战。我们提出了综合分类的多模态共同学习的基础上共同学习和相关的实现所面临的挑战。所采用的各种技术,包括最新的审查与一些应用程序和数据集。我们的最终目标是讨论挑战和观点以及未来工作的重要想法和方向,我们希望这些工作对专注于这个令人兴奋的领域的整个研究界有益。

英文摘要:

Multimodal deep learning systems which employ multiple modalities like text, image, audio, video, etc., are showing better performance in comparison with individual modalities (i.e., unimodal) systems. Multimodal machine learning involves multiple aspects: representation, translation, alignment, fusion, and co-learning. In the current state of multimodal machine learning, the assumptions are that all modalities are present, aligned, and noiseless during training and testing time. However, in real-world tasks, typically, it is observed that one or more modalities are missing, noisy, lacking annotated data, have unreliable labels, and are scarce in training or testing and or both. This challenge is addressed by a learning paradigm called multimodal co-learning. The modeling of a (resource-poor) modality is aided by exploiting knowledge from another (resource-rich) modality using transfer of knowledge between modalities, including their representations and predictive models. Co-learning being an emerging area, there are no dedicated reviews explicitly focusing on all challenges addressed by co-learning. To that end, in this work, we provide a comprehensive survey on the emerging area of multimodal co-learning that has not been explored in its entirety yet. We review implementations that overcome one or more co-learning challenges without explicitly considering them as co-learning challenges. We present the comprehensive taxonomy of multimodal co-learning based on the challenges addressed by co-learning and associated implementations. The various techniques employed to include the latest ones are reviewed along with some of the applications and datasets. Our final goal is to discuss challenges and perspectives along with the important ideas and directions for future work that we hope to be beneficial for the entire research community focusing on this exciting domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑