The integration of Federated Learning (FL) and Self-supervised Learning (SSL) offers a unique and synergetic combination to exploit the audio data for general-purpose audio understanding, without compromising user data privacy. However, rare efforts have been made to investigate the SSL models in the FL regime for general-purpose audio understanding, especially when the training data is generated by large-scale heterogeneous audio sources. In this paper, we evaluate the performance of feature-matching and predictive audio-SSL techniques when integrated into large-scale FL settings simulated with non-independently identically distributed (non-iid) data. We propose a novel Federated SSL (F-SSL) framework, dubbed FASSL, that enables learning intermediate feature representations from large-scale decentralized heterogeneous clients, holding unlabelled audio data. Our study has found that audio F-SSL approaches perform on par with the centralized audio-SSL approaches on the audio-retrieval task. Extensive experiments demonstrate the effectiveness and significance of FASSL as it assists in obtaining the optimal global model for state-of-the-art FL aggregation methods.
Institute for Advancing Intelligence, TCG Centres for Research and Education in Science and Technology(TCG科学与技术研究教育中心智能推进研究所);Indian Statistical Institute, Kolkata(印度统计研究所);Academy of Scientific and Innovative Research (AcSIR)(科学与创新研究院)
Federated Self-Supervised Learning for One-Shot Cross-Modal and Cross-Imaging Technique Segmentation
Hong Kong Baptist University(香港浸会大学);Narayana Superspeciality Hospital(纳拉亚纳超级专科医院);National Institute of Technology Durgapur(杜尔加布尔国家理工学院);Indian Institute of Technology(印度理工学院)
Zero-Shot Federated Learning with New Classes for Audio Classification
Global AI Accelerator, Ericsson(爱立信全球人工智能加速器);Ericsson Research(爱立信研究院)
联邦学习是一种从不同用户设备中提取见解同时保护用户隐私的有效方法。然而,具有完全不可见的数据分布的新类可以在联邦学习设置中的任何设备上流式传输,其数据不能被全局服务器或其他用户访问。为此,我们提出了一个统一的zero-shot框架来处理联邦学习过程中的这些挑战。我们在这里模拟了两种情况- 1)当用户没有报告新的类别标签时,使用传统的FL设置; 2)当用户报告新的类别标签时,我们通过计算与每个设备的新类别对应的类别相似性矩阵来合成Ancesterized Data Imperial,然后进行无监督聚类以区分不同用户的新类别。此外,我们提出的框架还可以处理参与用户的标签和模型中的统计异质性。我们在两个广泛使用的音频分类应用程序(关键字识别和城市声音分类)上,通过本地和全局更新中的新类别以及异构标签和模型,在不同的通信回合(FL迭代)中对我们的设备上框架进行了经验评估,并观察到平均确定性准确度分别增加了~4.041%和~4.258%。
School of Computer Science, McGill University(麦吉尔大学计算机科学学院);Mila - Quebec AI Institute(魁北克AI研究所);School of Mathematics, Physics and Computing, University of Southern Queensland(南方昆士兰大学数学、物理与计算学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);School of Science, Engineering and Environment, University of Salford(萨尔福德大学科学、工程与环境学院)
Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University and Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing(人工智能研究院、深圳MSU-BIT大学及粤港澳大湾区情感智能与普适计算联合实验室);School of Software Engineering, South China University of Technology(华南理工大学软件学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
Exploring bat song syllable representations in self-supervised audio encoders
Institute for Logic, Language and Computation, University of Amsterdam(阿姆斯特丹大学逻辑、语言与计算研究所);Museum für Naturkunde, Leibniz Institute for Evolution and Biodiversity Science(莱布尼茨进化与生物多样性科学自然博物馆)
AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmark
Fraunhofer Heinrich-Hertz-Institute(弗劳恩霍夫海因里希·赫兹研究所);Technische Universität Berlin(柏林工业大学);Korea University(高丽大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);BIFOLD – Berlin Institute for the Foundations of Learning and Data(BIFOLD——柏林学习与数据基础研究所)
Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics
University of Technology Sydney(悉尼科技大学);The University of Queensland(昆士兰大学);Futureverse;The Hong Kong University of Science and Technology(香港科技大学);CSIRO DATA61(联邦科学与工业研究组织数据61研究所);Netease Fuxi AI Lab(网易伏羲人工智能实验室)
Exploring Longitudinal Cough, Breath, and Voice Data for COVID-19 Progression Prediction via Sequential Deep Learning: Model Development and Validation
University of Cambridge(剑桥大学);University of Southampton(南安普顿大学)
BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
Communication Science Laboratories, Nippon Telegraph and Telephone Corporation(日本电报电话公司通信科学实验室)
预训练模型是现代机器学习系统中各个领域的特征提取器。在这项研究中,我们假设,表示有效的一般音频任务应该提供多个方面的强大功能的输入声音。为了识别声音而不管诸如变化的音高或音色的扰动,特征应该对这些扰动是鲁棒的。为了满足诸如情感或音乐类型识别等任务的不同需求,表征应该提供多方面的信息,例如局部和全局特征。为了实现我们的原则,我们提出了一种自监督学习方法:Bootstrap Your Own Latent(BYOL)for Audio(BYOL-A,发音为“viola”)。BYOL-A预训练对音频数据增强不变量的输入声音的表示,这使得学习的表示对声音的扰动具有鲁棒性。而BYOL-A编码器结合了局部和全局特征,并计算它们的统计信息,使表示提供多方面的信息。因此,学习的表示应该提供强大的和多方面的信息,以满足不同任务的各种需求。我们评估了BYOL-A与以前最先进的方法相比的一般音频任务性能,BYOL-A表现出了最佳平均结果72.4%和最佳VoxCeleb 1结果57.6%的泛化能力。大量的烧蚀实验表明,BYOL-A编码器架构有助于大部分性能,最后的关键部分诉诸BYOL框架和BYOL-A增强。我们的代码可以在https://github.com/nttcslab/byol-a上在线获得,以供将来研究。
Exploring Text-Queried Sound Event Detection with Audio Source Separation
School of Marine Science and Technology, Northwestern Polytechnical University(西北工业大学海洋科学与技术学院);Alibaba Group(阿里巴巴集团);Fortemedia Singapore(Fortemedia新加坡公司)
本文提出 TQ-SED 框架,通过预训练改进的 AudioSep-DP 语言查询音频源分离模型分离重叠事件,再由多分支检测目标声音,显著提升 SED 性能。
Exploring Multimodal Approaches for Alzheimer's Disease Detection Using Patient Speech Transcript and Audio Data
South China University of Technology(华南理工大学);University of Georgia(佐治亚大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital(麻省总医院)
Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Meituan(美团)
Exploring the User Experience of AI-Assisted Sound Searching Systems for Creative Workflows
Centre for Vision, Speech and Signal Processing, University of Surrey(视觉、语音和信号处理中心,萨里大学);Research and Development, British Broadcasting Corporation(英国广播公司研发部)
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
Hong Kong University of Science and Technology(香港科技大学);University of Science and Technology Beijing(北京科技大学);Hong Kong Baptist University(香港浸会大学);Microsoft(微软公司);Chinese University of Hong Kong(香港中文大学)
针对现有音频 LLM 编解码器因压缩设计导致的语义缺陷问题,提出 X-Codec 方法,通过在 RVQ 前后融入语义特征与损失,显著降低语音合成 WER 并扩展至非语音任务,提升音频生成性能。