Few-shot learning aims to train models that can recognize novel classes given just a handful of labeled examples, known as the support set. While the field has seen notable advances in recent years, they have often focused on multi-class image classification. Audio, in contrast, is often multi-label due to overlapping sounds, resulting in unique properties such as polyphony and signal-to-noise ratios (SNR). This leads to unanswered questions concerning the impact such audio properties may have on few-shot learning system design, performance, and human-computer interaction, as it is typically up to the user to collect and provide inference-time support set examples. We address these questions through a series of experiments designed to elucidate the answers to these questions. We introduce two novel datasets, FSD-MIX-CLIPS and FSD-MIX-SED, whose programmatic generation allows us to explore these questions systematically. Our experiments lead to audio-specific insights on few-shot learning, some of which are at odds with recent findings in the image domain: there is no best one-size-fits-all model, method, and support set selection criterion. Rather, it depends on the expected application scenario. Our code and data are available at https://github.com/wangyu/rethink-audio-fsl.
Rethinking Audio-visual Synchronization for Active Speaker Detection
Tsinghua University(清华大学);Institute for Artificial Intelligence, Tsinghua University (THUAI)(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology (BNRist)(北京信息科学与技术国家研究中心);University of Rochester(罗切斯特大学)
Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification
University of Kassel(卡塞尔大学);Fraunhofer IEE(弗劳恩霍夫研究所);Ghent University(根特大学);ML and Systems Biology, MPI of Biochemistry(生物化学Max Planck研究所);INRIA Montpellier(蒙彼利埃INRIA)
DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners
Jiutian Artificial Intelligence Research Institute, China Mobile, China State Key Laboratory of Multimedia Information Processing, Peking University, China(九天人工智能研究院,中国移动,国家多媒体信息处理重点实验室,北京大学)
RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Faculty of Psychology and Educational Sciences, Alexandru Ioan Cuza University of Iași(亚历山德鲁伊万库扎大学心理学与教育科学学院);School of Electronic Information, Wuhan University(武汉大学电子信息学院);School of Public Health, The University of Hong Kong(香港大学公共卫生学院);School of Computer Science, The University of Auckland(奥克兰大学计算机科学学院);Department of Data Science and Artificial Intelligence, Monash University(莫纳什大学数据科学与人工智能系)