EAT: Self-Supervised Pre-Training with Efficient Audio Transformer
MoE Key Lab of Artificial Intelligence, AI Institute（教育部人工智能重点实验室，人工智能研究院）；X-LANCE Lab, Department of Computer Science and Engineering（X-LANCE实验室，计算机科学与工程系）；Shanghai Jiao Tong University（上海交通大学）
Audio self-supervised learning (SSL) pre-training, which aims to learn good representations from unlabeled audio, has made remarkable progress. However, the extensive computational demands during pre-training pose a significant barrier to the potential application and optimization of audio SSL models. In this paper, inspired by the success of data2vec 2.0 in image modality and Audio-MAE in audio modality, we introduce Efficient Audio Transformer (EAT) to further improve the effectiveness and efficiency in audio SSL. The proposed EAT adopts the bootstrap self-supervised training paradigm to the audio domain. A novel Utterance-Frame Objective (UFO) is designed to enhance the modeling capability of acoustic events. Furthermore, we reveal that the masking strategy is critical in audio SSL pre-training, and superior audio representations can be obtained with large inverse block masks. Experiment results demonstrate that EAT achieves state-of-the-art (SOTA) performance on a range of audio-related tasks, including AudioSet (AS-2M, AS-20K), ESC-50, and SPC-2, along with a significant pre-training speedup up to ~15x compared to existing audio SSL models.
BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation
NTT Corporation（日本电信电话公司）
受计算机视觉自监督学习的最新进展的启发，我们探索了一种新的通用音频表示学习方法。我们建议从单个音频段学习通用音频表示，而不期望音频样本的不同时间段之间的关系。为了实现这一原则，我们引入了Bootstrap Your Own Latent（BYOL）for Audio（BYOL-A，发音为“viola”），这是一种基于BYOL的音频自监督学习方法，用于学习通用音频表示。与依赖于附近音频段的一致性或远程音频段的不一致性的大多数先前的音频自监督学习方法不同，BYOL-A在从单个音频段导出的增强音频段对中创建对比。通过标准化和增强技术的结合，BYOL-A在各种下游任务中实现了最先进的结果。广泛的消融研究也阐明了每个组件及其组合的贡献。
ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning
Laboratory of Cognitive Computing and Application, College of Intelligence and Computing（认知计算与应用实验室，智能与计算学院）；Huiyan Technology (Tianjin) Co., Ltd（慧颜科技（天津）有限公司）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）
MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning
University of Sheffield（谢菲尔德大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Queen Mary University of London（伦敦玛丽女王大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；Peking University（北京大学）；University of Tübingen（蒂宾根大学）；MPI-IS（马克斯·普朗克智能系统研究所）；Dartmouth College（达特茅斯学院）
DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection
Graduate School of Informatics, Kyoto University（京都大学情报学研究科）；National Institute of Advanced Industrial Science and Technology（独立行政法人产业技术综合研究所）；Faculty of Science and Engineering, Doshisha University（同志社大学理工学部）
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所台湾国际研究生计划社交网络与人本计算项目）；Department of Computer Science, National Chengchi University（国立政治大学资讯科学系）；Institute of Information Systems and Applications, National Tsing Hua University（国立清华大学资讯系统与应用研究所）；National Institute of Informatics（国立情报学研究所）；Department of Electrical Engineering and the Institute of Communications Engineering, National Tsing Hua University（国立清华大学电机工程学系与通讯工程研究所）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
An overview of neural architectures for self-supervised audio representation learning from masked spectrograms
Department of Electronic Systems, Aalborg University（电子系统系，奥尔堡大学）；Pioneer Centre for Artificial Intelligence（人工智能先锋中心）；National and Kapodistrian University of Athens（雅典国家与卡波蒂斯坦大学）
Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning
Academy for Engineering and Technology, Fudan University（复旦大学工程与应用技术研究院）；School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学计算机科学技术学院，上海市智能信息处理重点实验室）
SS-DPPN: A self-supervised dual-path foundation model for the generalizable cardiac audio representation
Department of Computer Science and Engineering（计算机科学与工程系）；BRAC University（布拉克大学）；Department of Electricial and Electronic Engineering（电气与电子工程系）；Department of Biomedical Engineering（生物医学工程系）；Linköping University（林肯堡大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；University of Maryland（马里兰大学）；School of Electrical Engineering, Computing and Mathematical Sciences（电气工程、计算与数学科学学院）
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Columbia University（哥伦比亚大学）；Google（谷歌）；Cornell University（康奈尔大学）
我们提出了一个框架，学习多模态表示从未标记的数据使用卷积自由Transformer架构。具体来说，我们的视频-音频-文本Transformer（VATT）将原始信号作为输入，并提取多模态表示，这些表示足够丰富，可以使各种下游任务受益。我们使用多模态对比损失从头开始训练VATT端到端，并通过视频动作识别、音频事件分类、图像分类和文本到视频检索等下游任务来评估其性能。此外，我们研究了一个模态不可知的，单骨干Transformer共享三个模态之间的权重。我们证明了无卷积VATT在下游任务中的性能优于最先进的基于ConvNet的架构。特别是，VATT的视觉Transformer在Kinetics-400上达到了82.1%的顶级准确率，在Kinetics-600上达到了83.6%，在Kinetics-700上达到了72.7%，在Moments in Time上达到了41.1%，这些都是新的记录，同时避免了监督式的预训练。转移到图像分类导致ImageNet上的top-1准确率为78.7%，而从头开始训练相同的Transformer的准确率为64.7%，这表明尽管视频和图像之间存在领域差距，但我们的模型具有普遍性。VATT的音频Transformer还在基于波形的音频事件识别方面创造了新纪录，在AudioSet上实现了39.4%的mAP，而无需任何监督的预训练。VATT的源代码是公开的。
Exploring bat song syllable representations in self-supervised audio encoders
Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Museum für Naturkunde, Leibniz Institute for Evolution and Biodiversity Science（莱布尼茨进化与生物多样性科学自然博物馆）
