Speech summarization has become an essential tool for efficiently managing and accessing the growing volume of spoken and audiovisual content. However, despite its increasing importance, speech summarization remains loosely defined. The field intersects with several research areas, including speech recognition, text summarization, and specific applications like meeting summarization. This survey not only examines existing datasets and evaluation protocols, which are crucial for assessing the quality of summarization approaches, but also synthesizes recent developments in the field, highlighting the shift from traditional systems to advanced models like fine-tuned cascaded architectures and end-to-end solutions. In doing so, we surface the ongoing challenges, such as the need for realistic evaluation benchmarks, multilingual datasets, and long-context handling.
A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition
Graduate School of Informatics, Nagoya University（国立 Nagoya 大学信息科学研究生院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Information Technology Center, Nagoya University（国立 Nagoya 大学信息技术中心）
Attention is a very popular and effective mechanism in artificial neural network-based sequence-to-sequence models. In this survey paper, a comprehensive review of the different attention models used in developing automatic speech recognition systems is provided. The paper focuses on the development and evolution of attention models for offline and streaming speech recognition within recurrent neural network- and Transformer- based architectures.
Predicting words and subword units (WSUs) as the output has shown to be effective for the attention-based encoder-decoder (AED) model in end-to-end speech recognition. However, as one input to the decoder recurrent neural network (RNN), each WSU embedding is learned independently through context and acoustic information in a purely data-driven fashion. Little effort has been made to explicitly model the morphological relationships among WSUs. In this work, we propose a novel character-aware (CA) AED model in which each WSU embedding is computed by summarizing the embeddings of its constituent …
Otto von Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）
Deep Learning based Automatic Speech Recognition (ASR) models are very successful, but hard to interpret. To gain better understanding of how Artificial Neural Networks (ANNs) accomplish their tasks, introspection methods have been proposed. Adapting such techniques from computer vision to speech recognition is not straight-forward, because speech data is more complex and less interpretable than image data. In this work, we introduce Gradient-adjusted Neuron Activation Profiles (GradNAPs) as means to interpret features and representations in Deep Neural Networks. GradNAPs are characteristic re…
A primary objective of news articles is to establish the factual record for an event, frequently achieved by conveying both the details of the specified event (i.e., the 5 Ws; Who, What, Where, When and Why regarding the event) and how people reacted to it (i.e., reported statements). However, existing work on news summarization almost exclusively focuses on the event details. In this work, we propose the novel task of summarizing the reactions of different speakers, as expressed by their reported statements, to a given event. To this end, we create a new multi-document summarization benchmark…
Queensland University of Technology（昆士兰理工大学）；Information Technology University（信息科技大学）；University of Lincoln（林肯大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；National University of Computer and Emerging Sciences（国家计算机与新兴科学大学）；Qatar University（卡塔尔大学）
Cross-Dataset, Age, and Gender Generalization: A Comprehensive Analysis of Fine-Tuning Strategies for Low-Resource Children's ASR
Department of Electronics and Communication Engineering, National Institute of Technology Sikkim, India（印度西西姆国立技术学院电子与通信工程系）；Signal Analysis and Interpretation Laboratory, University of Southern California, Los Angeles, USA（美国南加州大学洛杉矶分校信号分析与解释实验室）
We present a comprehensive study of deep bidirectional long short-term memory (LSTM) recurrent neural network (RNN) based acoustic models for automatic speech recognition (ASR). We study the effect of size and depth and train models of up to 8 layers. We investigate the training aspect and study different variants of optimization methods, batching, truncated backpropagation, different regularization techniques such as dropout and $L_2$ regularization, and different gradient clipping variants. The major part of the experimental analysis was performed on the Quaero corpus. Additional experiments…
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国）；Amazon GenAI（亚马逊生成人工智能）；Qatar Computing Research Institute（卡塔尔计算研究所）；University of Virginia（弗吉尼亚大学）
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）；School of Future Science and Engineering, Soochow University（苏州大学未来科学与工程学院）
Shahjalal University of Science & Technology（沙阿贾拉勒科学技术大学）
Hidden Markov model based various phoneme recognition methods for Bengali language is reviewed. Automatic phoneme recognition for Bengali language using multilayer neural network is reviewed. Usefulness of multilayer neural network over single layer neural network is discussed. Bangla phonetic feature table construction and enhancement for Bengali speech recognition is also discussed. Comparison among these methods is discussed.
Fédération des Caisses Desjardins du Québec（魁北克德雅尔丹储蓄信贷联盟）
This paper introduces a new synthesis-based defense algorithm for counteracting with a varieties of adversarial attacks developed for challenging the performance of the cutting-edge speech-to-text transcription systems. Our algorithm implements a Sobolev-based GAN and proposes a novel regularizer for effectively controlling over the functionality of the entire generative model, particularly the discriminator network during training. Our achieved results upon carrying out numerous experiments on the victim DeepSpeech, Kaldi, and Lingvo speech transcription systems corroborate the remarkable per…
This paper studies a novel pre-training technique with unpaired speech data, Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within a multi-task learning framework, we introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes, derived from an offline clustering model. One is to predict the pseudo codes via masked language modeling in encoder output, like HuBERT model, while the other lets the decoder learn to reconstruct pseudo codes autoregressively instead of generating textual scripts. In this way, the decoder learns to…
Modern topic identification (topic ID) systems for speech use automatic speech recognition (ASR) to produce speech transcripts, and perform supervised classification on such ASR outputs. However, under resource-limited conditions, the manually transcribed speech required to develop standard ASR systems can be severely limited or unavailable. In this paper, we investigate alternative unsupervised solutions to obtaining tokenizations of speech in terms of a vocabulary of automatically discovered word-like or phoneme-like units, without depending on the supervised training of ASR systems. Moreove…
Liverpool John Moores University（利物浦约翰摩尔斯大学）；Thapar Institute of Engineering and Technology（塔帕尔工程技术学院）
Investors make investment decisions depending on several factors such as fundamental analysis, technical analysis, and quantitative analysis. Another factor on which investors can make investment decisions is through sentiment analysis of news headlines, the sole purpose of this study. Natural Language Processing techniques are typically used to deal with such a large amount of data and get valuable information out of it. NLP algorithms convert raw text into numerical representations that machines can easily understand and interpret. This conversion can be done using various embedding techniqu…
Institute for Infocomm Research (I2R)（资讯通信研究所）；Changi General Hospital（樟宜综合医院）
Due to the lack of publicly available resources, conversation summarization has received far less attention than text summarization. As the purpose of conversations is to exchange information between at least two interlocutors, key information about a certain topic is often scattered and spanned across multiple utterances and turns from different speakers. This phenomenon is more pronounced during spoken conversations, where speech characteristics such as backchanneling and false-starts might interrupt the topical flow. Moreover, topic diffusion and (intra-utterance) topic drift are also more …
Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models
Tallinn University of Technology（塔林技术大学）；MBZUAI（穆罕默德·本·拉希德·阿勒马克图姆智能研究院）；Idiap Research Institute（伊迪普研究 institute）；Univ. Rennes, CNRS, Irisa（里昂大学，CNRS，Irisa）；Univ. Bretagne Sud, CNRS, Irisa（布列塔尼南部大学，CNRS，Irisa）；Validsoft Ltd.（Validsoft有限公司）
On The Landscape of Spoken Language Models: A Comprehensive Survey
Carnegie Mellon University（卡内基梅隆大学）；National Taiwan University（国立台湾大学）；Toyota Technological Institute at Chicago（丰田芝加哥技术研究所）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）；ENS - PSL, EHESS, CNRS（巴黎高等师范学院 - 巴黎文理研究大学、社会科学高等研究院、法国国家科学研究中心）
Google Research（谷歌研究院）；Grammarly；Megagon Labs（Megagon实验室）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Customer reviews are vital for making purchasing decisions in the Information Age. Such reviews can be automatically summarized to provide the user with an overview of opinions. In this tutorial, we present various aspects of opinion summarization that are useful for researchers and practitioners. First, we will introduce the task and major challenges. Then, we will present existing opinion summarization solutions, both pre-neural and neural. We will discuss how summarizers can be trained in the unsupervised, few-shot, and supervised regimes. Each regime has roots in different machine learning…
