The Modular Audio Recognition Framework (MARF) and its Applications: Scientific and Software Engineering Notes
Serguei A. Mokhov, Stephen Sinclair, Ian Clément, Dimitrios Nicolacopoulos
备注：
v2: add missing .ind file for index; 224 pages, 40 figures, 19 tables; index. A comprehensive description of AI and PR algorithms and data structures, software engineering design and implementation, and experiments. Source revision is maintained in the CVS at http://marf.sf.net
MARF is an open-source research platform and a collection of voice/sound/speech/text and natural language processing (NLP) algorithms written in Java and arranged into a modular and extensible framework facilitating addition of new algorithms. MARF can run distributively over the network and may act as a library in applications or be used as a source for learning and extension. A few example applications are provided to show how to use the framework. There is an API reference in the Javadoc format as well as this set of accompanying notes with the detailed description of the architectural design, algorithms, and applications. MARF and its applications are released under a BSD-style license and is hosted at SourceForge.net. This document provides the details and the insight on the internals of MARF and some of the mentioned applications.
Spooky Audio；University of Victoria（维多利亚大学）；McGill University（麦吉尔大学）
We release synth1B1, a multi-modal audio corpus consisting of 1 billion 4-second synthesized sounds, paired with the synthesis parameters used to generate them. The dataset is 100x larger than any audio dataset in the literature. We also introduce torchsynth, an open source modular synthesizer that generates the synth1B1 samples on-the-fly at 16200x faster than real-time (714MHz) on a single GPU. Finally, we release two new audio datasets: FM synth timbre and subtractive synth pitch. Using these datasets, we demonstrate new rank-based evaluation criteria for existing audio representations. Fin…
University College Dublin（都柏林大学学院）；University of Warwick（华威大学）
Audio quality assessment has been widely researched in the signal processing area. Full-reference objective metrics (e.g., POLQA, ViSQOL) have been developed to estimate the audio quality relying only on human rating experiments. To evaluate the audio quality of novel audio processing techniques, researchers constantly need to compare objective quality metrics. Testing different implementations of the same metric and evaluating new datasets are fundamental and ongoing iterative activities. In this paper, we present AQP - an open-source, node-based, light-weight Python pipeline for audio qualit…
SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence Shanghai Jiao Tong University（X-LANCE实验室，计算机科学学院，人工智能教育部重点实验室，上海交通大学）；Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）；Peng Cheng Laboratory（鹏城实验室）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Tianjin University（天津大学）；Hong Kong University of Science and Technology（香港科学大学）；Queen Mary University of London（伦敦玛丽女王大学）；Nanyang Technological University（南洋理工大学）；Shanghai Innovation Institute（上海创新研究院）
Speaker-attributed automatic speech recognition (SA-ASR) is a task to recognize "who spoke what" from multi-talker recordings. An SA-ASR system usually consists of multiple modules such as speech separation, speaker diarization and ASR. On the other hand, considering the joint optimization, an end-to-end (E2E) SA-ASR model has recently been proposed with promising results on simulation data. In this paper, we present our recent study on the comparison of such modular and joint approaches towards SA-ASR on real monaural recordings. We develop state-of-the-art SA-ASR systems for both modular and…
DeepFense: A Unified, Modular, and Extensible Framework for Robust Deepfake Audio Detection
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；University of Stuttgart（斯图加特大学）；National Institute of Informatics（国立信息学研究所）；Technical University of Berlin（柏林工业大学）
Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module
The University of Hong Kong, Fauclty of Science, Hong Kong（香港大学科学学院）；Aix-Marseille University, Laboratoire Parole et Langage (LPL), France（艾克斯-马赛大学语言与言语实验室（LPL））
MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
The way an object looks and sounds provide complementary reflections of its physical properties. In many settings cues from vision and audition arrive asynchronously but must be integrated, as when we hear an object dropped on the floor and then must find it. In this paper, we introduce a setting in which to study multi-modal object localization in 3D virtual environments. An object is dropped somewhere in a room. An embodied robot agent, equipped with a camera and microphone, must determine what object has been dropped -- and where -- by combining audio and visual signals with knowledge of th…
Electronic music artists and sound designers have unique workflow practices that necessitate specialized approaches for developing music information retrieval and creativity support tools. Furthermore, electronic music instruments, such as modular synthesizers, have near-infinite possibilities for sound creation and can be combined to create unique and complex audio paths. The process of discovering interesting sounds is often serendipitous and impossible to replicate. For this reason, many musicians in electronic genres record audio output at all times while they work in the studio. Subsequen…
ESPnet3: Infrastructure for Scalable Speech and Audio Research in the Foundation Model Era
Carnegie Mellon University（卡内基梅隆大学）；Brno University of Technology（布拉格技术大学）；Instituto Superior Técnico（技术高等研究院）；Hanyang University（翰阳大学）；Hitachi Astemo（日立阿美士多）；Shanghai Jiao Tong University（上海交通大学）
Hybrid and end-to-end (E2E) systems have their individual advantages, with different error patterns in the speech recognition results. By jointly modeling audio and text, the E2E model performs better in matched scenarios and scales well with a large amount of paired audio-text training data. The modularized hybrid model is easier for customization, and better to make use of a massive amount of unpaired text data. This paper proposes a two-pass hybrid and E2E cascading (HEC) framework to combine the hybrid and E2E model in order to take advantage of both sides, with hybrid in the first pass an…
Automatic speech recognition (ASR) of single channel far-field recordings with an unknown number of speakers is traditionally tackled by cascaded modules. Recent research shows that end-to-end (E2E) multi-speaker ASR models can achieve superior recognition accuracy compared to modular systems. However, these models do not ensure real-time applicability due to their dependency on full audio context. This work takes real-time applicability as the first priority in model design and addresses a few challenges in previous work on multi-speaker recurrent neural network transducer (MS-RNN-T). First, …
