使用多模态层级交叉注意力模型标注在线视频中的喜剧恶作剧内容
Labeling Comic Mischief Content in Online Videos with a Multimodal Hierarchical-Cross-Attention Model
AI总结:
提出端到端多模态 HICCAP 模型及视频、文本、音频数据集,用于喜剧恶作剧检测与类型分类,并在多个基准上验证其优于现有方法。
摘要:
我们致力于解决检测在线媒体中可疑内容的挑战,特别是喜剧恶作剧这一子类别。此类内容将暴力、成人内容或讽刺等元素与幽默相结合,因此难以检测。采用多模态方法对于捕捉喜剧恶作剧内容中固有的微妙细节至关重要。为解决这一问题,我们针对喜剧恶作剧检测任务提出了一种新颖的端到端多模态系统。作为贡献的一部分,我们发布了一个面向该目标任务的新数据集,包含三种模态:视频、文本(视频标题和字幕)以及音频。我们还设计了一个带标题的层级交叉注意力模型(HIerarchical Cross-attention model with CAPtions,HICCAP),以捕捉这些模态之间的复杂关系。结果表明,在喜剧恶作剧检测及其类型分类方面,所提方法相较于稳健基线和最先进模型取得了显著提升。这凸显了我们的系统在赋能用户方面的潜力,使他们能够对选择观看的在线内容作出知情决定。此外,我们在 UCF101、HMDB51 和 XD-Violence 数据集上进行了实验,将我们的模型与其他最先进方法进行比较,展示了所提模型在各种场景下的出色性能。
英文摘要:
We address the challenge of detecting questionable content in online media, specifically the subcategory of comic mischief. This type of content combines elements such as violence, adult content, or sarcasm with humor, making it difficult to detect. Employing a multimodal approach is vital to capture the subtle details inherent in comic mischief content. To tackle this problem, we propose a novel end-to-end multimodal system for the task of comic mischief detection. As part of this contribution, we release a novel dataset for the targeted task consisting of three modalities: video, text (video captions and subtitles), and audio. We also design a HIerarchical Cross-attention model with CAPtions (HICCAP) to capture the intricate relationships among these modalities. The results show that the proposed approach makes a significant improvement over robust baselines and state-of-the-art models for comic mischief detection and its type classification. This emphasizes the potential of our system to empower users, to make informed decisions about the online content they choose to see. In addition, we conduct experiments on the UCF101, HMDB51, and XD-Violence datasets, comparing our model against other state-of-the-art approaches showcasing the outstanding performance of our proposed model in various scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。