In this paper, we propose to pre-train audio encoders using synthetic patterns instead of real audio data. Our proposed framework consists of two key elements. The first one is Masked Autoencoder (MAE), a self-supervised learning framework that learns from reconstructing data from randomly masked counterparts. MAEs tend to focus on low-level information such as visual patterns and regularities within data. Therefore, it is unimportant what is portrayed in the input, whether it be images, audio mel-spectrograms, or even synthetic patterns. This leads to the second key element, which is synthetic data. Synthetic data, unlike real audio, is free from privacy and licensing infringement issues. By combining MAEs and synthetic patterns, our framework enables the model to learn generalized feature representations without real data, while addressing the issues related to real audio. To evaluate the efficacy of our framework, we conduct extensive experiments across a total of 13 audio tasks and 17 synthetic datasets. The experiments provide insights into which types of synthetic patterns are effective for audio. Our results demonstrate that our framework achieves performance comparable to models pre-trained on AudioSet-2M and partially outperforms image-based pre-training methods.
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所台湾国际研究生计划社交网络与人本计算项目）；Department of Computer Science, National Chengchi University（国立政治大学资讯科学系）；Institute of Information Systems and Applications, National Tsing Hua University（国立清华大学资讯系统与应用研究所）；National Institute of Informatics（国立情报学研究所）；Department of Electrical Engineering and the Institute of Communications Engineering, National Tsing Hua University（国立清华大学电机工程学系与通讯工程研究所）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
A Framework for Synthetic Audio Conversations Generation using Large Language Models
Innovative Cognitive Computing (IC2) Research Center（创新认知计算（IC2）研究中心）；School of Information Technology（信息科技学院）；King Mongkut's University of Technology Thonburi（泰国科技理工大学）
Generating Diverse Audio-Visual 360 Soundscapes for Sound Event Localization and Detection
University of Southern California（南加州大学）；Universidad Nacional Autónoma de México（墨西哥国立自治大学）；Universidad Autónoma de Tamaulipas（塔毛利帕斯自治大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）
Bootstrap an end-to-end ASR system by multilingual training, transfer learning, text-to-text mapping and synthetic audio
Alexa Speech, Amazon Development Centre（Alexa语音，亚马逊开发中心）；Alexa Speech, Amazon Development Center GmbH（Alexa语音，亚马逊开发中心有限公司）；Alexa Speech, Amazon Development Center（Alexa语音，亚马逊开发中心）
Adaptive re-calibration of channel-wise features for Adversarial Audio Classification
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
与DeepFake图像和视频不同，DeepFake音频从检测角度进行的探索相对较少，并且现有的合成语音分类解决方案要么使用复杂网络，要么不推广到使用不同的生成和优化方法获得的不同种类的合成语音。通过这项工作，我们提出了一种使用注意力特征融合进行合成语音检测的通道重新校准特征，并将其性能与不同的检测方法进行比较，包括End2End模型和基于Resnet的模型，这些模型使用文本到语音和声码器系统（如WaveNet，WaveRNN，Tactotron和WaveGlow）生成合成语音。我们还在Resnet模型中使用挤压激发（SE）块进行了实验，发现这种组合能够获得更好的性能。除了分析，我们还证明了线性频率倒谱系数（LFCC）和梅尔频率倒谱系数（MFCC）的组合使用的注意力特征融合技术创建更好的输入特征表示，可以帮助更简单的模型概括以及合成语音分类任务。我们的模型（基于Resnet，使用特征融合）在Fake or Real（FoR）数据集上进行训练，并能够在FoR数据上实现95%的测试准确度，并且在调整此框架后，我们使用不同的生成模型生成的样本的平均准确度为90%。
End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding
Integrative Sciences and Engineering Programme, NUS Graduate School（新加坡国立大学研究生院整合科学与工程项目）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
Lightweight Model Attribution and Detection of Synthetic Speech via Audio Residual Fingerprints
Faculty of Computer Science, Ruhr University Bochum（计算机科学系，博尔库姆鲁尔大学）；Electronic Systems of Medical Engineering, Technische Universität Berlin（医学工程电子系统，柏林技术大学）
Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models
National University of Singapore（新加坡国立大学）；ASUS Intelligent Cloud Services（华硕智能云服务）；I2R, Agency for Science, Technology, and Research (A*STAR)（新加坡科技研究局信息研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Universidad Politécnica de Madrid（马德里理工大学）
Nagoya University, Japan（名古屋大学）；Nankai University, China（南开大学）；Meta, USA（Meta公司）；National Institute of Information and Communications Technology, Japan（日本信息通信技术研究所）
XSQ-AST: An Explainable Audio Spectrogram Transformer Framework for Localising Synthetic Speech Artifacts
AudioLab, School of Physics, Engineering and Technology, University of York, United Kingdom（英国约克大学物理工程技术学院音频实验室）；Department of Computer Science, University of York, United Kingdom（英国约克大学计算机科学系）；SEED -- Electronic Arts (EA), Sweden（瑞典艺电SEED部门）；Electronic Arts (EA), United States（美国艺电）
Generative Deep Learning and Signal Processing for Data Augmentation of Cardiac Auscultation Signals: Improving Model Robustness Using Synthetic Audio
School of Electrical Engineering, Computing, and Mathematical Sciences (EECMS), Faculty of Science and Engineering, Curtin University（电气工程、计算与数学科学学院（EECMS），科学与工程学院， Curtin 大学）
LLM-Based Synthetic Ground Truth Generation for Audio-Based Emotion Classification via In-Context Learning
School of Business, Technical University of Applied Sciences Augsburg（应用技术大学阿沙芬堡商学院）；Data Science und Autonome Systeme Technologietransferzentrum (TTZ)（数据科学与自主系统技术转移中心（TTZ））
