Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes

使用视觉-语言模型检测和缓解多模态模因中的仇恨内容

Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

本文利用视觉-语言模型（VLMs）解决多模态模因中仇恨内容的检测与缓解问题，提出定义引导提示技术和统一框架UnHateMeme，通过替换仇恨组件将其转化为非仇恨形式，并在LLaVA等模型上验证了有效性。

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

Memes on the Internet are often harmless and sometimes amusing. However, by using certain types of images, text, or combinations of both, the seemingly harmless meme becomes a multimodal type of hate speech -- a hateful meme. The Hateful Memes Challenge is a first-of-its-kind competition which focuses on detecting hate speech in multimodal memes and it proposes a new data set containing 10,000+ new examples of multimodal content. We utilize VisualBERT -- which meant to be the BERT of vision and language -- that was trained multimodally on images and captions and apply Ensemble Learning. Our ap…

Caption Enriched Samples for Improving Hateful Memes Detection

The recently introduced hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Specifically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Motivated by the need to model the contrast between the image content and the overlayed text, we suggest applying an off-the-shelf image captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the…

视觉-语言模型综述及其在仇恨模因挑战中的表现

A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge

本文综述了多种视觉-语言模型在仇恨模因挑战中的表现，对比早期融合与晚期融合方法，发现早期融合模型（尤其CLIP）显著优于晚期融合，AUROC达70.06。

鲁棒适应大规模多模态模型用于检索增强的仇恨表情包检测

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

本文提出了一种鲁棒适应框架，用于提升大规模多模态模型在检索增强的仇恨表情包检测中的性能与泛化能力，同时提高模型的可解释性。

MemeIntel: 可解释性地检测宣传性及仇恨言论的迷因

MemeIntel: Explainable Detection of Propagandistic and Hateful Memes

MemeIntel通过多阶段优化方法和视觉-语言模型，提升了阿拉伯语宣传性迷因和英语仇恨言论的检测与解释生成性能，准确率提升显著。