The remarkable generative capabilities of denoising diffusion models have raised new concerns regarding the authenticity of the images we see every day on the Internet. However, the vast majority of existing deepfake detection models are tested against previous generative approaches (e.g. GAN) and usually provide only a "fake" or "real" label per image. We believe a more informative output would be to augment the per-image label with a localization map indicating which regions of the input have been manipulated. To this end, we frame this task as a weakly-supervised localization problem and identify three main categories of methods (based on either explanations, local scores or attention), which we compare on an equal footing by using the Xception network as the common backbone architecture. We provide a careful analysis of all the main factors that parameterize the design space: choice of method, type of supervision, dataset and generator used in the creation of manipulated images; our study is enabled by constructing datasets in which only one of the components is varied. Our results show that weakly-supervised localization is attainable, with the best performing detection method (based on local scores) being less sensitive to the looser supervision than to the mismatch in terms of dataset or generator.
Inconsistency-aware Multimodal Schrödinger Bridge for Deepfake Localization
Department of Computer Science and Techonology, Huaqiao University（华侨大学计算机科学与技术系）；Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Huaqiao University（厦门计算机视觉与模式识别重点实验室）；Tongji University（同济大学）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）
Deepfakes have become a critical social problem, and detecting them is of utmost importance. Also, deepfake generation methods are advancing, and it is becoming harder to detect. While many deepfake detection models can detect different types of deepfakes separately, they perform poorly on generalizing the detection performance over multiple types of deepfake. This motivates us to develop a generalized model to detect different types of deepfakes. Therefore, in this work, we introduce a practical digital forensic tool to detect different types of deepfakes simultaneously and propose Transfer l…
LAVA: Layered Audio-Visual Anti-tampering Watermarking for Robust Deepfake Detection and Localization
School of Computer Science and Engineering（计算机科学与工程学院）；UNSW Sydney（新南威尔士大学悉尼分校）；School of Information and Communication Technology（信息与通信技术学院）；Griffith University（格里菲斯大学）
DDL: A Large-Scale Dataset for Deepfake Detection and Localization in Diversified Real-World Scenarios
AntGroup（蚂蚁集团）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hefei University of Technology（合肥工业大学）；Anhui Province Key Laboratory of Digital Security（安徽省数字安全重点实验室）；A ⋆ STAR Centre for Frontier AI Research（A ⋆ STAR前沿人工智能研究中心）
Monash University（莫纳什大学）；Flinders University（弗林德斯大学）；Curtin University（科廷大学）；Qualcomm（高通公司）；Queen Mary University of London（伦敦大学玛丽皇后学院）；American University of Sharjah（沙迦美国大学）
All in One: Unifying Deepfake Detection, Tampering Localization, and Source Tracing with a Robust Landmark-Identity Watermark
School of Computer Science and Technology, Xinjiang University, Urumqi, China（新疆大学计算机科学与技术学院）；Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, Urumqi, China（新疆多模态智能处理与信息安全工程技术研究中心）
Morphology-optimized Multi-Scale Fusion: Combining Local Artifacts and Mesoscopic Semantics for Deepfake Detection and Localization
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
University of Macau（澳门大学）；Macau University of Science and Technology（澳门科技大学）
With the continuous research on Deepfake forensics, recent studies have attempted to provide the fine-grained localization of forgeries, in addition to the coarse classification at the video-level. However, the detection and localization performance of existing Deepfake forensic methods still have plenty of room for further improvement. In this work, we propose a Spatial-Temporal Deepfake Detection and Localization (ST-DDL) network that simultaneously explores spatial and temporal features for detecting and localizing forged regions. Specifically, we design a new Anchor-Mesh Motion (AMM) algor…
Delocate: Detection and Localization for Deepfake Videos with Randomly-Located Tampered Traces
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与通信工程学院）；Show Lab, National University of Singapore（新加坡国立大学Show实验室）；Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University（南洋理工大学快速丰富目标搜索实验室）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络安全学院）
本文提出 Delocate 模型，通过随机遮挡 ROI 的人脸恢复与伪造掩码监督定位两阶段方法，同时提升未知域 Deepfake 视频检测和篡改区域定位性能。
Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；Monash University（莫纳什大学）
We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, eg, loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as an aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss…
SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model
University of Liverpool, UK（利物浦大学）；Nanyang Technological University, SG（南洋理工大学）；WMG, University of Warwick（沃里克大学工程学院）；The Chinese University of Hong Kong, Shenzhen, Guangdong, China（香港中文大学（深圳））
Whether, Which, and Whose: Solving the Triple Challenge of Deepfake Proactive Forensics in Multi-Face Scenarios
College of Computer Science and Technology（计算机科学与技术学院）；Xinjiang University（新疆大学）；College of Computer Science and Electronic Engineering（计算机科学与电子工程学院）；Hunan University（湖南大学）
Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning
School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University（计算机科学与工程学院、信息科技教育部重点实验室、广东省信息安全技术重点实验室、中山大学）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）
Omni-Fake: Benchmarking Unified Multimodal Social Media Deepfake Detection
University of Liverpool（利物浦大学）；University of Exeter（埃克塞特大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）
Addressing Deepfake Issue in Selfie banking through camera based authentication
Metallurgical And Materials Engineering Indian Institute Of Technology Kharagpur, India（冶金与材料工程 印度理工学院 阁agalpur分校）；Information Systems Carnegie Mellon University Doha, Qatar（信息系统 卡内基梅隆大学 多哈分校）
MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models
School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University, Guangzhou 510006, China（计算机科学与工程学院，信息技术MOE实验室，广东省信息安全技术重点实验室，中山大学，广州510006，中国）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；Department of Computer and Information Science, University of Macau.（计算机与信息科学系，澳门大学）
We propose `Hide-and-Seek', a weakly-supervised framework that aims to improve object localization in images and action localization in videos. Most existing weakly-supervised methods localize only the most discriminative parts of an object rather than all relevant parts, which leads to suboptimal performance. Our key idea is to hide patches in a training image randomly, forcing the network to seek other relevant parts when the most discriminative part is hidden. Our approach only needs to modify the input image and can work with any network designed for object localization. During testing, we…
Nanjing University（南京大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Modern deep learning models require large amounts of accurately annotated data, which is often difficult to satisfy. Hence, weakly supervised tasks, including weakly supervised object localization~(WSOL) and detection~(WSOD), have recently received attention in the computer vision community. In this paper, we motivate and propose the weakly supervised foreground learning (WSFL) task by showing that both WSOL and WSOD can be greatly improved if groundtruth foreground masks are available. More importantly, we propose a complete WSFL pipeline with low computational cost, which generates pseudo bo…
Recent advances of deep learning have achieved remarkable performances in various challenging computer vision tasks. Especially in object localization, deep convolutional neural networks outperform traditional approaches based on extraction of data/task-driven features instead of hand-crafted features. Although location information of region-of-interests (ROIs) gives good prior for object localization, it requires heavy annotation efforts from human resources. Thus a weakly supervised framework for object localization is introduced. The term "weakly" means that this framework only uses image-l…
Weakly-supervised object localization methods tend to fail for object classes that consistently co-occur with the same background elements, e.g. trains on tracks. We propose a method to overcome these failures by adding a very small amount of model-specific additional annotation. The main idea is to cluster a deep network's mid-level representations and assign object or distractor labels to each cluster. Experiments show substantially improved localization results on the challenging ILSVC2014 dataset for bounding box detection and the PASCAL VOC2012 dataset for semantic segmentation.
National Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；Nanjing University（南京大学）
Weakly supervised object localization (WSOL) aims to localize objects with only image-level labels. Previous methods often try to utilize feature maps and classification weights to localize objects using image level annotations indirectly. In this paper, we demonstrate that weakly supervised object localization should be divided into two parts: class-agnostic object localization and object classification. For class-agnostic object localization, we should use class-agnostic methods to generate noisy pseudo annotations and then perform bounding box regression on them without class labels. We pro…
We propose an improved technique for weakly-supervised object localization. Conventional methods have a limitation that they focus only on most discriminative parts of the target objects. The recent study addressed this issue and resolved this limitation by augmenting the training data for less discriminative parts. To this end, we employ an effective data augmentation for improving the accuracy of the object localization. In addition, we introduce improved learning techniques by optimizing Convolutional Neural Networks (CNN) based on the state-of-the-art model. Based on extensive experiments,…
Seoul National University（首尔国立大学）；LG AI Research（LG人工智能研究院）
Weakly supervised object localization aims to find a target object region in a given image with only weak supervision, such as image-level labels. Most existing methods use a class activation map (CAM) to generate a localization map; however, a CAM identifies only the most discriminative parts of a target object rather than the entire object region. In this work, we find the gap between classification and localization in terms of the misalignment of the directions between an input feature and a class-specific weight. We demonstrate that the misalignment suppresses the activation of CAM in area…
The Hong Kong University of Science and Technology（香港科技大学）
Despite deep convolutional neural networks boost the performance of image classification and segmentation in digital pathology analysis, they are usually weak in interpretability for clinical applications or require heavy annotations to achieve object localization. To overcome this problem, we propose a weakly supervised learning-based approach that can effectively learn to localize the discriminative evidence for a diagnostic label from weakly labeled training data. Experimental results show that our proposed method can reliably pinpoint the location of cancerous evidence supporting the decis…
Shanghai Jiao Tong University（上海交通大学）；Shanghai Frontier Science Research Center for Gravitational Wave Detection（上海引力波探测前沿科学研究中心）
Pseudo-supervised learning methods have been shown to be effective for weakly supervised object localization tasks. However, the effectiveness depends on the powerful regularization ability of deep neural networks. Based on the assumption that the localization network should have similar location predictions on different versions of the same image, we propose a two-stage approach to learn more consistent localization. In the first stage, we propose a mask-based pseudo label generator algorithm, and use the pseudo-supervised learning method to initialize an object localization network. In the s…
