Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schne… 展开作者
Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schneider, Rickmer Schulte, Karol Urbanczyk, Jann Goschenhofer, Christian Heumann, Rasmus Hvingelby, Daniel Schalk, Matthias Aßenmacher
This book is the result of a seminar in which we reviewed multimodal approaches and attempted to create a solid overview of the field, starting with the current state-of-the-art approaches in the two subfields of Deep Learning individually. Further, modeling frameworks are discussed where one modality is transformed into the other, as well as models in which one modality is utilized to enhance representation learning for the other. To conclude the second part, architectures with a focus on handling both modalities simultaneously are introduced. Finally, we also cover other modalities as well as general-purpose multi-modal models, which are able to handle different tasks on different modalities within one unified architecture. One interesting application (Generative Art) eventually caps off this booklet.
Supervised Deep Multimodal Matrix Factorization for Interpretable Brain Network Analysis
Dept. of Mathematics & Operational Research University of Mons(数学与运筹学系蒙斯大学);Dept. of Computer Science & Engineering Lehigh University(计算机科学与工程系莱斯大学);Dept. of Industrial & Systems Engineering Lehigh University(工业与系统工程系莱斯大学)
Deep Learning and Machine Learning, Advancing Big Data Analytics and Management: Handy Appetizer
AppCubic;University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Department of Electrical and Computer Engineering, University of Hawaii at Manoa(夏威夷大学马诺а分校电子与计算机工程系);Indiana University(印第安纳大学);Georgia Institute of Technology(佐治亚理工学院);Purdue University(普渡大学);Kyoto University(京都大学);Rutgers University(罗格斯大学);National Taiwan Normal University(台湾师范大学);Emory University(埃默里大学);Sichuan University(四川大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学)
Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation
National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);University of Edinburgh(爱丁堡大学);Beijing Institute of Technology(北京理工大学)
人口贩运是一种全球性流行病,影响到全球数百万人。人口贩运的主要形式-性贩运-显著增加,主要是因为伴游网站大量存在,人口贩运者可以在这些网站上公开刊登任意伴游广告。在本文中,我们在自动检测涉嫌涉及人口贩运的广告方面迈出了重要的一步。我们提出了一个名为Trafficking-10 k的新数据集,其中包含超过10,000个针对此任务的广告注释。该数据集包含每个广告的两个信息源:文本和图像。为了准确检测贩运广告,我们设计并训练了一个名为Human Trafficking Deep Network(HTDN)的深度多模态模型。
Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
Bielefeld University(比勒费尔德大学);University of Bonn(波恩大学)
互联网上的模因通常是无害的,有时很有趣。然而,通过使用某些类型的图像,文本或两者的组合,看似无害的模因变成了一种多模态的仇恨言论-仇恨模因。Hateful Memes Challenge是首个专注于检测多模态模因中的仇恨言论的竞赛,它提出了一个包含10,000多个多模态内容新示例的新数据集。我们利用VisualBERT --这意味着是视觉和语言的BERT--它是在图像和字幕上进行多模态训练的,并应用了包围学习。我们的方法在挑战测试集上达到了0.811 AUROC,准确率为0.765,并在仇恨模因挑战的3,173名参与者中排名第三。