Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Michigan（密歇根大学）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Stanford University（斯坦福大学）
Jeff Da, Maxwell Forbes, Rowan Zellers, Anthony Zheng, Jena D. Hwang, Antoine Bosselut, Yejin Choi
Multimodal disinformation, from 'deepfakes' to simple edits that deceive, is an important societal problem. Yet at the same time, the vast majority of media edits are harmless -- such as a filtered vacation photo. The difference between this example, and harmful edits that spread disinformation, is one of intent. Recognizing and describing this intent is a major challenge for today's AI systems.
We present the task of Edited Media Understanding, requiring models to answer open-ended questions that capture the intent and implications of an image edit. We introduce a dataset for our task, EMU, with 48k question-answer pairs written in rich natural language. We evaluate a wide variety of vision-and-language models for our task, and introduce a new model PELICAN, which builds upon recent progress in pretrained multimodal representations. Our model obtains promising results on our dataset, with humans rating its answers as accurate 40.35% of the time. At the same time, there is still much work to be done -- humans prefer human-annotated captions 93.56% of the time -- and we provide analysis that highlights areas for further progress.
HateMirage: An Explainable Multi-Dimensional Dataset for Decoding Faux Hate and Subtle Online Abuse
Indian Institute of Information Technology Dharwad（印度信息技术学院达尔瓦德）；Manipal Institute of Technology, Manipal Academy of Higher Education（曼普及高等教育学院技术学院）；Indraprastha Institute of Information Technology Delhi（印度普拉斯塔信息技术学院德里）
Why Misinformation is Created? Detecting them by Integrating Intent Features
Jilin University（吉林大学）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Liaoning Normal University（辽宁师范大学）；University of Shanghai for Science and Technology（上海理工大学）
Harmfully Manipulated Images Matter in Multimodal Misinformation Detection
Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education, Jilin University（吉林大学 教育部符号计算与知识工程重点实验室）；College of Computer Science and Technology, Jilin University（吉林大学 计算机科学与技术学院）
Reasoning About the Unsaid: Misinformation Detection with Omission-Aware Graph Inference
Media Synthesis and Forensics Lab, Institute of Computing Technology, Chinese Academy of Sciences（媒体合成与取证实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
University of Washington（华盛顿大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Carnegie Mellon University（卡内基梅隆大学）
Even to a simple and short news headline, readers react in a multitude of ways: cognitively (e.g. inferring the writer's intent), emotionally (e.g. feeling distrust), and behaviorally (e.g. sharing the news with their friends). Such reactions are instantaneous and yet complex, as they rely on factors that go beyond interpreting factual content of news. We propose Misinfo Reaction Frames (MRF), a pragmatic formalism for modeling how readers might react to a news headline. In contrast to categorical schema, our free-text dimensions provide a more nuanced way of understanding intent beyond being …
DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning
Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Lab of AI Safety, Beijing, China（人工智能安全国家重点实验室，北京，中国）；University of Chinese Academy of Sciences, CAS, Beijing, China（中国科学院大学，中国科学院，北京，中国）；Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China（复杂系统认知与决策智能重点实验室，自动化研究所，中国科学院，北京，中国）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（人工智能学院，中国科学院大学，北京，中国）
