The recent generative AI models' capability of creating realistic and human-like content is significantly transforming the ways in which people communicate, create and work. The machine-generated content is a double-edged sword. On one hand, it can benefit the society when used appropriately. On the other hand, it may mislead people, posing threats to the society, especially when mixed together with natural content created by humans. Hence, there is an urgent need to develop effective methods to detect machine-generated content. However, the lack of aligned multimodal datasets inhibited the development of such methods, particularly in triple-modality settings (e.g., text, image, and voice). In this paper, we introduce RU-AI, a new large-scale multimodal dataset for robust and effective detection of machine-generated content in text, image and voice. Our dataset is constructed on the basis of three large publicly available datasets: Flickr8K, COCO and Places205, by adding their corresponding AI duplicates, resulting in a total of 1,475,370 instances. In addition, we created an additional noise variant of the dataset for testing the robustness of detection models. We conducted extensive experiments with the current SOTA detection methods on our dataset. The results reveal that existing models still struggle to achieve accurate and robust detection on our dataset. We hope that this new data set can promote research in the field of machine-generated content detection, fostering the responsible use of generative AI. The source code and datasets are available at https://github.com/ZhihaoZhang97/RU-AI.
A Multimodal Approach to Predict Social Media Popularity
Nirma Institute of Technology(尼玛理工学院);Delhi Technological University(德里理工大学);UALR(阿肯色大学小石城分校);National University of Singapore(新加坡国立大学);IIIT-Delhi(德里英迪拉·甘地信息技术学院)
多个模态表示数据源传递信息的不同方面。现代社交媒体平台是多模态数据的主要来源之一,其中用户通过发布文本以及多媒体内容(例如图像和视频)来使用不同的表达模式以共享信息。这些帖子中包含的多模式信息可能有助于预测其受欢迎程度。据我们所知,不存在这样的多模态数据集来预测社交媒体照片。在这项工作中,我们提出了一个多模态数据集consistiting的内容,上下文和社会信息的流行预测。具体来说,我们在ACM Multimedia Grand Challenge 2017中使用图像内容,标题,描述和标签来增强SMPT 1数据集的社交媒体预测。接下来,在本文中,我们提出了一种利用视觉特征(即,内容信息),文本特征(即,上下文信息),以及社交特征(例如,平均视图和组计数)来预测社交媒体照片在视图计数方面的流行度。实验结果证实,尽管我们的多模态方法使用了来自SMP-T1的训练数据集的一半,但它实现了与最先进的方法相当的性能。
YTClickbait21K: Human-Annotated Multimodal Dataset for YouTube Clickbait Detection Across Diverse Channels and Content Categories
Department of Computer Science and Engineering, Rajshahi University of Engineering & Technology(拉贾沙希工程与技术大学计算机科学与工程系);Department of Electrical Engineering, Qatar University(卡塔尔大学电气工程系);Department of Civil and Environmental Engineering, Qatar University(卡塔尔大学土木与环境工程系);SenseNet Inc.(SenseNet公司)
Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Content
Mathematics and Statistics Department University of Texas at Dallas(德克萨斯大学达拉斯分校数学与统计学系);Computer Science Department University of Texas at Dallas(德克萨斯大学达拉斯分校计算机科学系);Naveen Jindal School of Management University of Texas at Dallas(德克萨斯大学达拉斯分校奈文·金达尔管理学院)
Revisiting Content-Based Music Recommendation: Efficient Feature Aggregation from Large-Scale Music Models
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
A Context-aware Attention and Graph Neural Network-based Multimodal Framework for Misogyny Detection
Department of Computer Science and Engineering(计算机科学与工程系);Indian Institute of Technology Indore(印度理工学院印德分校);Department of Computer Science and Engineering, National Institute of Technology Srinagar(计算机科学与工程系,印度国立技术学院斯里 Nagar分校)
New Ideas and Trends in Deep Multimodal Content Understanding: A Review
LIACS, Leiden University(莱顿大学高级计算机科学研究所);College of Systems Engineering, NUDT(国防科技大学系统工程学院);Center for Machine Vision and Signal Analysis, University of Oulu(奥卢大学机器视觉与信号分析中心)
Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline
School of Computer Science and Information Engineering, Hefei University of Technology(计算机科学与信息工程学院,合肥工业大学);School of Computer Science and Technology, Northwestern Polytechnical University(计算机科学与技术学院,西北工业大学)
Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
The University of Sydney, School of Computer Science(悉尼大学计算机科学学院);University of Technology Sydney, School of Computer Science(新南威尔士大学技术学院);University of Technology Sydney, TD School(新南威尔士大学TD学院);Australian Catholic University, Peter Faber Business School(澳大利亚天主教大学彼得·法伯商学院)
Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
LMU Munich(慕尼黑大学);Siemens AG(西门子公司);Technical University of Munich(慕尼黑工业大学);University of Oxford(牛津大学);University of California, Santa Barbara(加州大学圣巴巴拉分校)
MultiHateLoc: Towards Temporal Localisation of Multimodal Hate Content in Online Videos
Department of Computer Science, University of Exeter(埃克塞特大学计算机科学系);Institute for Analytics and Data Science, University of Essex(埃塞克斯大学分析与数据科学研究所);School of Computer Science, University of Birmingham(伯明翰大学计算机科学学院)
Detecting and Explaining Fake News Short Videos with Multimodal Content and Real-World Evidence
AI and Social Good Lab, AI Media Centre(AI媒体中心人工智能与社会公益实验室);Southern University of Science and Technology(南方科技大学);Beijing Normal University(北京师范大学);Hong Kong Baptist University(香港浸会大学)