Active speaker detection in videos addresses associating a source face, visible in the video frames, with the underlying speech in the audio modality. The two primary sources of information to derive such a speech-face relationship are i) visual activity and its interaction with the speech signal and ii) co-occurrences of speakers' identities across modalities in the form of face and speech. The two approaches have their limitations: the audio-visual activity models get confused with other frequently occurring vocal activities, such as laughing and chewing, while the speakers' identity-based methods are limited to videos having enough disambiguating information to establish a speech-face association. Since the two approaches are independent, we investigate their complementary nature in this work. We propose a novel unsupervised framework to guide the speakers' cross-modal identity association with the audio-visual activity for active speaker detection. Through experiments on entertainment media videos from two benchmark datasets, the AVA active speaker (movies) and Visual Person Clustering Dataset (TV shows), we show that a simple late fusion of the two approaches enhances the active speaker detection performance.
Rethinking Audio-visual Synchronization for Active Speaker Detection
Tsinghua University（清华大学）；Institute for Artificial Intelligence, Tsinghua University (THUAI)（清华大学人工智能研究院）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）；University of Rochester（罗切斯特大学）
CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction
University Hospital rechts der Isar, Technical University of Munich（慕尼黑工业大学右岸医院）；School of Artificial Intelligence, the Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）人工智能学院）；Department of Computer Science and Technology, University of Science and Technology Beijing（北京科技大学计算机科学与技术系）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；Guangdong Provincial Key Laboratory of Big Data Computing（广东省大数据计算重点实验室）；Shenzhen Research Institute of Big data（深圳大数据研究院）；University of Bremen（不来梅大学）；Imperial College London（帝国理工学院）
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China（中国科学技术大学国家语音语言信息处理工程研究中心）；iFlytek（科大讯飞）；School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）
Speaker Recognition in Realistic Scenario Using Multimodal Data
University of Engineering and Technology Taxila（塔克西拉工程技术大学）；Swarm Robotics Lab NCRA（国家机器人与自动化研究所群体机器人实验室）；Deutsches Elektronen-Synchrotron DESY（德国电子同步加速器研究所）
RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Dalian Maritime University（大连海事大学）
PIAVE: A Pose-Invariant Audio-Visual Speaker Extraction Network
Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；National University of Singapore（新加坡国立大学）
Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings
Speech and Hearing (SPandH) group, School of Computer Science, The University of Sheffield, Sheffield, United Kingdom（语音与听力（SPandH）小组，计算机科学学院，谢菲尔德大学，谢菲尔德，英国）；South Westphalia University of Applied Sciences, Iserlohn, Germany（西南弗劳恩霍夫应用科学大学，伊塞尔洛恩，德国）
MeMo: Attentional Momentum for Real-Time Audio-Visual Target Speaker Extraction Under Impaired Visual Conditions
Department of Electrical and Electronic Engineering, Faculty of Engineering, The Hong Kong Polytechnic University（电子工程系，工程学院，香港理工大学）；Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong（系统工程与工程管理系，香港中文大学）；School of Artificial Intelligence (SAI), The Chinese University of Hong Kong, Shenzhen（人工智能学院（SAI），香港中文大学深圳校区）；School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学）；Tongyi Lab, Alibaba Group, Singapore（通义实验室，阿里巴巴集团，新加坡）
