Alzheimer's Disease (AD) is nowadays the most common form of dementia, and its automatic detection can help to identify symptoms at early stages, so that preventive actions can be carried out. Moreover, non-intrusive techniques based on spoken data are crucial for the development of AD automatic detection systems. In this light, this paper is presented as a contribution to the ADReSS Challenge, aiming at improving AD automatic detection from spontaneous speech. To this end, recordings from 108 participants, which are age-, gender-, and AD condition-balanced, have been used as training set to perform two different tasks: classification into AD/non-AD conditions, and regression over the Mini-Mental State Examination (MMSE) scores. Both tasks have been performed extracting 28 features from speech -- based on prosody and voice quality -- and 51 features from the transcriptions -- based on lexical and turn-taking information. Our results achieved up to 87.5 % of classification accuracy using a Random Forest classifier, and 4.54 of RMSE using a linear regression with stochastic gradient descent over the provided test set. This shows promising results in the automatic detection of Alzheimer's Disease through speech and lexical features.
We present a probabilistic model that uses both prosodic and lexical cues for the automatic segmentation of speech into topically coherent units. We propose two methods for combining lexical and prosodic information using hidden Markov models and decision trees. Lexical information is obtained from a speech recognizer, and prosodic features are extracted automatically from speech waveforms. We evaluate our approach on the Broadcast News corpus, using the DARPA-TDT evaluation metrics. Results show that the prosodic model alone is competitive with word-based segmentation methods. Furthermore, we…
University of Washington（华盛顿大学）；Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）
In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses. For automatically parsing spoken utterances, we introduce a model that integrates transcribed text and acoustic-prosodic features using a convolutional neural network over energy and pitch trajectories coupled with an attention-based recurrent neural network that accepts text and prosodic features. We find that different types of acoustic-prosodic features are individually helpful, and together give statistically significant improvements in parse and disfluency detection F1 scores ove…
CLASH: Counterfactual Auditing of Lexical and Prosodic Reliance in Spoken Sarcasm Detection
Imperial College London（伦敦帝国理工学院）；New York University（纽约大学）；Technical University of Munich（慕尼黑工业大学）；Tencent Inc.（腾讯公司）；Wuhan University（武汉大学）；Nankai University（南开大学）
ProsAudit, a prosodic benchmark for self-supervised speech models
PSL Research University（巴黎文理研究大学）；ENS（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；EHESS（法国高等社会科学研究院）；INRIA（法国国家信息与自动化研究所）；Université Paris Cité（巴黎西岱大学）；Bielefeld University（比勒费尔德大学）；INRAE（法国国家农业、食品与环境研究院）；Institut Agro（农业研究院）；Meta AI Research（Meta人工智能研究院）
Queen Mary University of London（伦敦玛丽女王大学）；Jožef Stefan Institute（约热夫·斯特凡研究所）
We present two multimodal fusion-based deep learning models that consume ASR transcribed speech and acoustic data simultaneously to classify whether a speaker in a structured diagnostic task has Alzheimer's Disease and to what degree, evaluating the ADReSSo challenge 2021 data. Our best model, a BiLSTM with highway layers using words, word probabilities, disfluency features, pause information, and a variety of acoustic features, achieves an accuracy of 84% and RSME error prediction of 4.26 on MMSE cognitive scores. While predicting cognitive decline is more challenging, our models show improve…
Impact of automatic speech recognition quality on Alzheimer's disease detection from spontaneous speech: a reproducible benchmark study with lexical modeling and statistical validation
LLM-Anchored Paralinguistic Enrichment for Alzheimer's Disease Detection
Tianjin University（天津大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Fuzhou University（福州大学）；Huiyan Technology (Tianjin) Co., Ltd（慧眼科技（天津）有限公司）
University of Washington（华盛顿大学）；University of Minnesota（明尼苏达大学）
In recent years there has been a burgeoning interest in the use of computational methods to distinguish between elicited speech samples produced by patients with dementia, and those from healthy controls. The difference between perplexity estimates from two neural language models (LMs) - one trained on transcripts of speech produced by healthy participants and the other trained on transcripts from patients with dementia - as a single feature for diagnostic classification of unseen transcripts has been shown to produce state-of-the-art performance. However, little is known about why this approa…
The present study has two goals relating to the grammar of prosody, understood as the rhythms and melodies of speech. First, an overview is provided of the computable grammatical and phonetic approaches to prosody analysis which use hypothetico-deductive methods and are based on learned hermeneutic intuitions about language. Second, a proposal is presented for an inductive grounding in the physical signal, in which prosodic structure is inferred using a language-independent method from the low-frequency spectrum of the speech signal. The overview includes a discussion of computational aspects …
Universitat Pompeu Fabra（庞培法布拉大学）；Universitat de Barcelona（巴塞罗那大学）
Alzheimer's Disease is the most common form of dementia. Automatic detection from speech could help to identify symptoms at early stages, so that preventive actions can be carried out. This research is a contribution to the ADReSSo Challenge, we analyze the usage of a SotA ASR system to transcribe participant's spoken descriptions from a picture. We analyse the loss of performance regarding the use of human transcriptions (measured using transcriptions from the 2020 ADReSS Challenge). Furthermore, we study the influence of a language model -- which tends to correct non-standard sequences of wo…
A crucial step in processing speech audio data for information extraction, topic detection, or browsing/playback is to segment the input into sentence and topic units. Speech segmentation is challenging, since the cues typically present for segmenting text (headers, paragraphs, punctuation) are absent in spoken language. We investigate the use of prosody (information gleaned from the timing and melody of speech) for these tasks. Using decision tree and hidden Markov modeling techniques, we combine prosodic cues with word-based approaches, and evaluate performance on two speech corpora, Broadca…
People convey information extremely effectively through spoken interaction using multiple channels of information transmission: the lexical channel of what is said, and the non-lexical channel of how it is said. We propose studying human perception of spoken communication as a means to better understand how information is encoded across these channels, focusing on the question 'What characteristics of communicative context affect listener's expectations of speech?'. To investigate this, we present a novel behavioural task testing whether listeners can discriminate between the true utterance in…
Involvement hot spots have been proposed as a useful concept for meeting analysis and studied off and on for over 15 years. These are regions of meetings that are marked by high participant involvement, as judged by human annotators. However, prior work was either not conducted in a formal machine learning setting, or focused on only a subset of possible meeting features or downstream applications (such as summarization). In this paper we investigate to what extent various acoustic, linguistic and pragmatic aspects of the meetings, both in isolation and jointly, can help detect hot spots. In t…
We describe a statistical approach for modeling dialogue acts in conversational speech, i.e., speech-act-like units such as Statement, Question, Backchannel, Agreement, Disagreement, and Apology. Our model detects and predicts dialogue acts based on lexical, collocational, and prosodic cues, as well as on the discourse coherence of the dialogue act sequence. The dialogue model is based on treating the discourse structure of a conversation as a hidden Markov model and the individual dialogue acts as observations emanating from the model states. Constraints on the likely sequence of dialogue act…
The detection of perceived prominence in speech has attracted approaches ranging from the design of linguistic knowledge-based acoustic features to the automatic feature learning from suprasegmental attributes such as pitch and intensity contours. We present here, in contrast, a system that operates directly on segmented speech waveforms to learn features relevant to prominent word detection for children's oral fluency assessment. The chosen CRNN (convolutional recurrent neural network) framework, incorporating both word-level features and sequence information, is found to benefit from the per…
Beyond surface form: A pipeline for semantic analysis in Alzheimer's Disease detection from spontaneous speech
Healthy Lifespan Institute, The University of Sheffield, Sheffield, United Kingdom（谢菲尔德大学健康寿命研究所）；School of Computer Science, The University of Sheffield, Sheffield, United Kingdom（谢菲尔德大学计算机科学学院）；Institute of Data Science and Artificial Intelligence, The University of Exeter, Exeter, United Kingdom（埃克塞特大学数据科学与人工智能研究所）；Faculty of Linguistics, Pontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre, Brazil（里约格朗德杜斯尔格斯天主教大学语言学院）；Bioinformatics Multidisciplinary Environment, Digital Metropolis Institute, Federal University of Rio Grande do Norte, Natal, Brazil（巴西国家联邦大学里约格朗德多北大学生物信息学跨学科环境）；Institute of Physics, Federal University of Rio Grande do Sul, Porto Alegre, Brazil（巴西联邦大学里约格朗德多南大学物理研究所）
In this paper, we have introduced and evaluated intonation based feature for scoring the English speech of nonnative English speakers in Indian context. For this, we created an automated spoken English scoring engine to learn from the manual evaluation of spoken English. This involved using an existing Automatic Speech Recognition (ASR) engine to convert the speech to text. Thereafter, macro features like accuracy, fluency and prosodic features were used to build a scoring model. In the process, we introduced SimIntonation, short for similarity between spoken intonation pattern and "ideal" i.e…
Explainable and Generalisable LLM-based Cognitive Decline Detection with Spontaneous Speech
Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Peking University Sixth Hospital（北京大学第六医院）；Peking University（北京大学）；OPPO Research Institute（OPPO研究院）；Peking University Third Hospital（北京大学第三医院）
A Methodology for Explainable Large Language Models with Integrated Gradients and Linguistic Analysis in Text Classification
Bioinformatics Multidisciplinary Environment (BioME)（生物信息学多学科环境（BioME））；Digital Metropolis Institute (IMD)（数字大都市研究所（IMD））；Federal University of Rio Grande do Norte (UFRN)（北里奥格兰德联邦大学（UFRN））；Mobile Brain（移动大脑（公司））；Institute of Psychiatry (IPUB)（精神病学研究所（IPUB））；Federal University of Rio de Janeiro (UFRJ)（里约热内卢联邦大学（UFRJ））；The University of Exeter（埃克塞特大学）；Institute for Data Science and Artificial Intelligence at the University of Exeter（埃克塞特大学数据科学与人工智能研究所）；The University of Sheffield（谢菲尔德大学）；Pontifical Catholic University of Rio Grande do Sul (PUCRS)（南里奥格兰德天主教大学（PUCRS））
