Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning
Johannes Kepler University（约翰内斯·开普勒大学）；Aalto University（阿alto大学）；University of Michigan-Flint（密歇根大学弗林特分校）；Macquarie University（麦考瑞大学）；Institute of Computational Perception, Johannes Kepler University Linz（计算感知研究所，约翰内斯·开普勒大学林茨）；Linz Institute of Technology（林茨技术研究所）
Felix Breiteneder, Mohammad Belal, Muhammad Saad Saeed, Shahed Masoudian, Usman Naseem, Kulshrestha Juhi, Markus Schedl, Shah Nawaz
Internet memes are powerful tools for communication, capable of spreading political, psychological, and sociocultural ideas. However, they can be harmful and can be used to disseminate hate toward targeted individuals or groups. Although previous studies have focused on designing new detection methods, these often rely on modal-complete data, such as text and images. In real-world settings, however, modalities like text may be missing due to issues like poor OCR quality, making existing methods sensitive to missing information and leading to performance deterioration. To address this gap, in this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of harmful meme detection methods in the presence of modal-incomplete data. Specifically, we propose a new baseline method that learns a shared representation for multiple modalities by projecting them independently. These shared representations can then be leveraged when data is modal-incomplete. Experimental results on two benchmark datasets demonstrate that our method outperforms existing approaches when text is missing. Moreover, these results suggest that our method allows for better integration of visual features, reducing dependence on text and improving robustness in scenarios where textual information is missing. Our work represents a significant step forward in enabling the real-world application of harmful meme detection, particularly in situations where a modality is absent.
All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproduction
State Key Laboratory of Complex System Modeling and Simulation Technology（复杂系统建模与仿真技术国家重点实验室）；Science and Technology on Integrated Information System Laboratory Institute of Software Chinese Academy of Sciences（软件研究所信息集成系统技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement
University of Electronic Science and Technology of China（电子科技大学）；Southwestern University of Finance and Economics（西南财经大学）；Intelligent Digital Media Technology Key Laboratory of Sichuan Province（四川省智能数字媒体技术重点实验室）
Detecting and Understanding Harmful Memes: A Survey
IIIT-Delhi（德里印度理工学院）；Wipro AI Labs（威普罗人工智能实验室）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）；Sofia University（索非亚大学）；University of Padova（帕多瓦大学）；Facebook AI（脸书人工智能）；Sapienza University of Rome（罗马第一大学）
Detecting the Role of an Entity in Harmful Memes: Techniques and Their Limitations
BJIT Limited（BJIT有限公司）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
随着时间的推移，有害或滥用的在线内容不断增加，引起了社交媒体平台，政府机构和政策制定者的关注。这种有害或滥用的内容可能对社会产生重大的负面影响，例如，网络欺凌可能导致自杀，关于COVID-19的谣言可能导致疫苗犹豫，推广COVID-19的假治疗可能导致健康损害和死亡。在线发布和共享的内容可以是文本的、视觉的或两者的组合，例如，in a meme模因.在这里，我们描述了我们在检测有害模因中实体（英雄，恶棍，受害者）角色的实验，这是约束-2022共享任务的一部分，以及我们的任务系统。我们进一步提供了不同实验设置的比较分析（即，单峰、多峰、注意力和增强）。为了可重复性，我们公开了我们的实验代码。\url{https://github.com/robi56/harmful_memes_block_fusion}
DISARM: Detecting the Victims Targeted by Harmful Memes
Indraprastha Institute of Information Technology - Delhi（德里英德拉普拉斯塔信息技术学院）；Wipro AI Labs（威普罗人工智能实验室）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Content
Mathematics and Statistics Department University of Texas at Dallas（德克萨斯大学达拉斯分校数学与统计学系）；Computer Science Department University of Texas at Dallas（德克萨斯大学达拉斯分校计算机科学系）；Naveen Jindal School of Management University of Texas at Dallas（德克萨斯大学达拉斯分校奈文·金达尔管理学院）
Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset
Oxford Internet Institute（牛津互联网研究院）；Dept. of Computer Science（计算机科学系）；Oxford Uehiro Centre for Practical Ethics（牛津上广实践伦理中心）；Dept. of Engineering Science（工程科学系）
模因被用于通过社交网络传播思想。虽然大多数模因是为了幽默而创造的，但有些模因在图片和文字的结合下变得令人讨厌。自动检测仇恨模因可以帮助减少其有害的社会影响。与传统的视觉和文本信息在语义上一致的多模态任务不同，仇恨模因检测的挑战在于其独特的多模态信息。模因中的图像和文本是弱对齐的，甚至是不相关的，这要求模型理解内容并在多个模态上执行推理。在本文中，我们专注于多模态仇恨模因检测，并提出了一种新的方法，将图像描述过程中的模因检测过程。我们在多模态模因数据集上进行了广泛的实验，并说明了我们的方法的有效性。我们的模型在Hateful Memes Detection Challenge上取得了令人鼓舞的结果。
Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology Taxila（塔克西拉工程大学）；Dalhousie University（达尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（校际微电子中心）；Johannes Kepler University Linz（约翰开普勒林茨大学）；Linz Institute of Technology（林茨理工学院）
Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities
IMT School for Advanced Studies of Lucca（卢卡高等研究院）；Institute of Computational Perception, Johannes Kepler University（约翰开普勒大学计算感知研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology, Taxila（塔克西拉工程技术大学）；Dalhousie University（戴尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（微电子学研究中心）；Linz Institute of Technology（林茨理工学院）
Sinogram upsampling using Primal-Dual UNet for undersampled CT and radial MRI reconstruction
Otto von Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）；Human Technopole（人类技术园）；German Centre for Neurodegenerative Disease（德国神经退行性疾病中心）；Centre for Behavioural Brain Sciences（行为脑科学中心）
