arXiv 2505.190102025-07-30cs.CVcs.CL

Co-AttenDWG：面向多模态攻击性内容检测的协同注意力维度门控与专家融合

Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection

Department of Computer Science and Engineering, Bangladesh University of Business and Technology（计算机科学与工程系，孟加拉国商业技术大学）；Department of Computer Science, American International University-Bangladesh（计算机科学系，美国国际大学-孟加拉国）

Md. Mithun Hossain, Md. Shakil Hossain, Sudipto Chaki, M. F. Mridha

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态攻击性内容检测中跨模态交互不足、融合策略僵化的问题，提出带维度门控的协同注意力与专家融合模型Co-AttenDWG，在两个数据集上实现了当前最优性能。

摘要：

多模态学习已成为关键研究方向，因为融合文本与视觉信息可大幅提升分类、检索、场景理解等任务的性能。尽管大型预训练模型已取得进展，但现有方法往往存在跨模态交互不足、融合策略僵化的问题，无法充分利用不同模态的互补优势。为解决这些局限，我们提出Co-AttenDWG，即带维度门控的协同注意力与专家融合方法。该方法首先将文本和视觉特征投影到共享嵌入空间，通过专门的协同注意力机制实现模态间同步的细粒度交互；维度门控网络进一步在通道层面自适应调制特征贡献，以突出显著信息。同时，双路径编码器独立优化模态专属表示，额外的交叉注意力层进一步对齐模态。最终特征通过专家融合模块聚合，该模块融合了习得的门控与自注意力机制，生成鲁棒的统一表示。在MIMIC和SemEval Memotion 1.0数据集上的实验结果表明，Co-AttenDWG取得了当前最优性能和更优的跨模态对齐效果，凸显了其在各类多模态应用中的有效性。

英文摘要：

Multi-modal learning has emerged as a crucial research direction, as integrating textual and visual information can substantially enhance performance in tasks such as classification, retrieval, and scene understanding. Despite advances with large pre-trained models, existing approaches often suffer from insufficient cross-modal interactions and rigid fusion strategies, failing to fully harness the complementary strengths of different modalities. To address these limitations, we propose Co-AttenDWG, co-attention with dimension-wise gating, and expert fusion. Our approach first projects textual and visual features into a shared embedding space, where a dedicated co-attention mechanism enables simultaneous, fine-grained interactions between modalities. This is further strengthened by a dimension-wise gating network, which adaptively modulates feature contributions at the channel level to emphasize salient information. In parallel, dual-path encoders independently refine modality-specific representations, while an additional cross-attention layer aligns the modalities further. The resulting features are aggregated via an expert fusion module that integrates learned gating and self-attention, yielding a robust unified representation. Experimental results on the MIMIC and SemEval Memotion 1.0 datasets show that Co-AttenDWG achieves state-of-the-art performance and superior cross-modal alignment, highlighting its effectiveness for diverse multi-modal applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。