Coarse-to-Fine Proposal Refinement Framework for Audio Temporal Forgery Detection and Localization
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；Alibaba Group（阿里巴巴集团）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间科学与技术学院）
Recently, a novel form of audio partial forgery has posed challenges to its forensics, requiring advanced countermeasures to detect subtle forgery manipulations within long-duration audio. However, existing countermeasures still serve a classification purpose and fail to perform meaningful analysis of the start and end timestamps of partial forgery segments. To address this challenge, we introduce a novel coarse-to-fine proposal refinement framework (CFPRF) that incorporates a frame-level detection network (FDN) and a proposal refinement network (PRN) for audio temporal forgery detection and localization. Specifically, the FDN aims to mine informative inconsistency cues between real and fake frames to obtain discriminative features that are beneficial for roughly indicating forgery regions. The PRN is responsible for predicting confidence scores and regression offsets to refine the coarse-grained proposals derived from the FDN. To learn robust discriminative features, we devise a difference-aware feature learning (DAFL) module guided by contrastive representation learning to enlarge the sensitive differences between different frames induced by minor manipulations. We further design a boundary-aware feature enhancement (BAFE) module to capture the contextual information of multiple transition boundaries and guide the interaction between boundary information and temporal features via a cross-attention mechanism. Extensive experiments show that our CFPRF achieves state-of-the-art performance on various datasets, including LAV-DF, ASVS2019PS, and HAD.
Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network
MoE Key Laboratory of Information Technology（信息科技联合实验室）；Sun Yat-sen University（中山大学）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；Alibaba Group（阿里巴巴集团）
Towards Generalizable Deepfake Detection via Forgery-aware Audio-Visual Adaptation: A Variational Bayesian Approach
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Department of New Networks, Pengcheng Laboratory（鹏城实验室网络部）；School of Cyber Science and Technology, Sun Yat-sen University（中山大学网络安全科学与技术学院）；School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院）；School of Cyberspace Science, Harbin Institute of Technology（哈尔滨工业大学网络空间科学学院）
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；Shenzhen University（深圳大学）
Image editing techniques enable people to modify the content of an image without leaving visual traces and thus may cause serious security risks. Hence the detection and localization of these forgeries become quite necessary and challenging. Furthermore, unlike other tasks with extensive data, there is usually a lack of annotated forged images for training due to annotation difficulties. In this paper, we propose a self-adversarial training strategy and a reliable coarse-to-fine network that utilizes a self-attention mechanism to localize forged regions in forgery images. The self-attention mo…
Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI
Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与类脑智能技术重点实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
Context-aware TFL: A Universal Context-aware Contrastive Learning Framework for Temporal Forgery Localization
School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University（计算机科学与工程学院、信息科技关键实验室、广东信息安全技术重点实验室、中山大学）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；School of Cyber Science and Technology, Shenzhen Campus, Sun Yat-sen University（网络安全科学与技术学院、深圳校区、中山大学）
From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection
Center for Future Media, School of Computer Science and Engineering, University of Electronic Science and Technology of China（未来媒体中心，计算机科学与工程学院，电子科技大学）
AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（社会网络与以人为本计算计划、台湾国际研究生计划、中国科学院）；Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所、国立清华大学）；Department of Computer Science, National Chengchi University（计算机科学系、国立成功大学）；Research Center for Information Technology Innovation, Academia Sinica（信息技术创新研究中心、中国科学院）；Institute of Information Science, Academia Sinica（信息科学研究所、中国科学院）
A Unified Framework for Modality-Agnostic Deepfakes Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；University at Buffalo, State University of New York（纽约州立大学布法罗分校）；RealAI Inc（RealAI公司）
DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University at Buffalo, State University of New York（纽约州立大学布法罗分校）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；Monash University（莫纳什大学）
We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, eg, loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as an aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss…
AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（学术.sinica.edu.tw 社会网络与人本计算计划，台湾国际研究生计划，台湾.sinica.edu.tw）；Department of Computer Science, National Chengchi University（计算机科学系，国立政治大学）；Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所，国立清华大学）；Research Center for Information Technology Innovation, Academia Sinica（资讯技术创新研究中心，学术.sinica.edu.tw）；Institute of Information Science, Academia Sinica（资讯研究所，学术.sinica.edu.tw）
Identity-Driven Multimedia Forgery Detection via Reference Assistance
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（复旦大学计算机科学与技术学院）；CEC GienTech Technology Co.,Ltd.（上海国家数字视听工程技术研究中心吉恩特科技有限公司）
RFM-Editing 2: Text-Guided Audio Editing with Rectified Flow Matching and Coarse-to-Fine Diffusion Transformers
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉、语音与信号处理中心）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Fisheries College, Ocean University of China（中国海洋大学水产学院）；College of Information and Electrical Engineering, China Agricultural University（中国农业大学信息与电气工程学院）
Audio splicing is one of the most common manipulation techniques in the area of audio forensics. In this paper, the magnitudes of acoustic channel impulse response and ambient noise are proposed as the environmental signature. Specifically, the spliced audio segments are detected according to the magnitude correlation between the query frames and reference frames via a statically optimal threshold. The detection accuracy is further refined by comparing the adjacent frames. The effectiveness of the proposed method is tested on two data sets. One is generated from TIMIT database, and the other o…
Previous works on scene classification are mainly based on audio or visual signals, while humans perceive the environmental scenes through multiple senses. Recent studies on audio-visual scene classification separately fine-tune the largescale audio and image pre-trained models on the target dataset, then either fuse the intermediate representations of the audio model and the visual model, or fuse the coarse-grained decision of both models at the clip level. Such methods ignore the detailed audio events and visual objects in audio-visual scenes (AVS), while humans often identify different scen…
