先对齐再注意：对齐视觉与文本特征以进行多模态仇恨内容检测
Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection
AI总结：
本文提出一种上下文感知注意框架，通过结合注意层在融合前对齐视觉与文本特征，以解决多模态仇恨内容检测中传统融合技术无法有效关注特定模态特征及多语言支持不足的问题，在MUTE和MultiOFF基准上F1分数分别达69.7%和70.3%，性能优于现有系统。
摘要：
多模态仇恨内容检测是一项具有挑战性的任务，需要在视觉和文本模态之间进行复杂的推理。因此，创建一个通过中间融合有效捕捉视觉与文本特征相互作用的有意义的多模态表示至关重要。传统的融合技术无法有效关注特定模态特征。此外，大多数研究仅集中于英语而忽略了其他低资源语言。本文提出了一个用于多模态仇恨内容检测的上下文感知注意框架，并针对英语和非英语语言对其进行了评估。所提出的方法结合了一个注意层，以有意义地对齐视觉和文本特征。这种对齐使得在融合它们之前能够选择性地关注特定模态特征。我们在两个基准仇恨模因数据集上评估了所提出的方法，即MUTE（孟加拉语码混合）和MultiOFF（英语）。评估结果证明了我们提出方法的有效性，在MUTE和MultiOFF数据集上的F1分数分别为69.7%和70.3%。这些分数表明，在这些数据集上，相较于最先进的系统，性能提升了约2.5%和3.2%。我们的实现可在https://github.com/eftekhar-hossain/Bengali-Hateful-Memes获取。
英文摘要：
Multimodal hateful content detection is a challenging task that requires complex reasoning across visual and textual modalities. Therefore, creating a meaningful multimodal representation that effectively captures the interplay between visual and textual features through intermediate fusion is critical. Conventional fusion techniques are unable to attend to the modality-specific features effectively. Moreover, most studies exclusively concentrated on English and overlooked other low-resource languages. This paper proposes a context-aware attention framework for multimodal hateful content detection and assesses it for both English and non-English languages. The proposed approach incorporates an attention layer to meaningfully align the visual and textual features. This alignment enables selective focus on modality-specific features before fusing them. We evaluate the proposed approach on two benchmark hateful meme datasets, viz. MUTE (Bengali code-mixed) and MultiOFF (English). Evaluation results demonstrate our proposed approach's effectiveness with F1-scores of $69.7$% and $70.3$% for the MUTE and MultiOFF datasets. The scores show approximately $2.5$% and $3.2$% performance improvement over the state-of-the-art systems on these datasets. Our implementation is available at https://github.com/eftekhar-hossain/Bengali-Hateful-Memes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
