Neural network-based speaker recognition has achieved significant improvement in recent years. A robust speaker representation learns meaningful knowledge from both hard and easy samples in the training set to achieve good performance. However, noisy samples (i.e., with wrong labels) in the training set induce confusion and cause the network to learn the incorrect representation. In this paper, we propose a two-step audio-visual deep cleansing framework to eliminate the effect of noisy labels in speaker representation learning. This framework contains a coarse-grained cleansing step to search for the peculiar samples, followed by a fine-grained cleansing step to filter out the noisy labels. Our study starts from an efficient audio-visual speaker recognition system, which achieves a close to perfect equal-error-rate (EER) of 0.01\%, 0.07\% and 0.13\% on the Vox-O, E and H test sets. With the proposed multi-modal cleansing mechanism, four different speaker recognition networks achieve an average improvement of 5.9\%. Code has been made available at: \textcolor{magenta}{\url{https://github.com/TaoRuijie/AVCleanse}}.
Self-Supervised Training of Speaker Encoder with Multi-Modal Diverse Positive Pairs
National University of Singapore(新加坡国立大学);Fortemedia(富迪媒体);University of Eastern Finland(东芬兰大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Bremen(不来梅大学);Kriston AI(Kriston人工智能)
MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues
The Hong Kong Polytechnic University(香港理工大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Posts and Telecommunications(北京邮电大学);Dalian Maritime University(大连海事大学)
M3ANet: Multi-scale and Multi-Modal Alignment Network for Brain-Assisted Target Speaker Extraction
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);Alibaba group(阿里巴巴集团);Department of Automation, Tsinghua University(清华大学自动化系);Beijing National Research Center for lnformation Science and Technology, Tsinghua University(清华大学信息科学与技术国家研究中心)
Cross-modal Speaker Verification and Recognition: A Multilingual Perspective
Istituto Italiano di Tecnologia (IIT)(意大利技术研究院);University of Insubria(因苏布里亚大学);University of Engineering and Technology Taxila(塔克西拉工程技术大学);Information Technology University(信息技术大学)
Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities
University of Science and Technology Beijing(北京科技大学);Beijing Institute of Technology(北京理工大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Alibaba Group(阿里巴巴集团);Zhejiang Institute of Quality Sciences(浙江省质量科学研究院);Tianjin University(天津大学);Technical University of Munich(慕尼黑工业大学)
Robust Audio-Visual Target Speaker Extraction with Emotion-Aware Multiple Enrollment Fusion
School of Computer Science, Wuhan University, Wuhan, China(1 武汉大学计算机学院,武汉,中国);School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China(2 香港中文大学(深圳)人工智能学院,中国);School of Artificial Intelligence, Wuhan University, Wuhan, China(3 武汉大学人工智能学院,武汉,中国);School of Cyber Science and Engineering, Wuhan University, Wuhan, China(4 武汉大学网络科学与工程学院,武汉,中国);Digital Innovation Research Center, Duke Kunshan University, Kunshan, China(5 香港中文大学(深圳)数字创新研究中心,中国);AI Center, OPPO, Beijing, China(6 OPPO人工智能中心,北京,中国)
Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition
The Chinese University of Hong Kong(香港中文大学);National Research Council Canada(加拿大国家研究委员会);The Hong Kong Polytechnic University(香港理工大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario
Northwestern Polytechnical University(西北工业大学);Microsoft Corporation(微软公司);Beijing Shell Shell Technology Co., Ltd.(北京贝壳科技有限公司);University of Science and Technology of China(中国科学技术大学)
Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge
Xiamen University(厦门大学);Beijing Jiaotong University(北京交通大学);School of Electronic Science and Engineering(电子科学与技术学院);School of Electronic Information(电子信息学院);School of Informatics(信息学院)
SpeakerCard-1M: An Evidence-Grounded Corpus for In-the-Wild Speaker Verification
Brno University of Technology, Czechia(布拉格技术大学,捷克);Peking University, China(北京大学,中国);Xiaomi, China(小米,中国);Athens University of Economics and Business, Greece(雅典经济与商业大学,希腊);The Hong Kong Polytechnic University, Hong Kong(香港理工大学,香港);Nanjing University, China(南京大学,中国)
CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization
Hujing Digital Media and Entertainment Group(华景数字媒体与娱乐集团);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS部门);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
Unsupervised active speaker detection in media content using cross-modal information
University of Southern California(南加州大学)
我们提出了一个跨模态的无监督框架,主动说话人检测的媒体内容,如电视节目和电影。机器学习的进步使得从语音和面部图像中识别个人的性能令人印象深刻。我们利用说话人身份信息的语音和面孔,并制定积极的说话人检测作为一个语音人脸分配任务,使积极的说话人的脸和基本的语音识别相同的人(字符)。我们表示的语音段在其相关的扬声器的身份距离,从所有其他语音段,捕捉一个相对的身份结构的视频。然后,我们分配一个积极的发言人的脸,每个语音段从同时出现的脸,使所获得的一组积极的发言人的脸显示一个类似的相对身份结构。此外,我们提出了一个简单而有效的方法来解决语音段扬声器存在屏幕外。我们在三个基准数据集上评估了所提出的系统- Visual Person Clustering数据集,AVA活跃扬声器数据集和Columbia数据集-由来自娱乐和广播媒体的视频组成,并显示出与最先进的完全监督方法相比具有竞争力的性能。
CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction
The Chinese University of Hong Kong(香港中文大学);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Vocal Engineering Technologies Limited(声动科技有限公司)