本研究评估了三种大型语言模型聊天机器人（ChatGPT-3.5、ChatGPT-4 和 Bard）利用自发语音文本识别阿尔茨海默病与认知正常个体的能力，发现 Bard 在识别 AD 上表现最佳，GPT-4 在识别 CN 上最优，但均未达到临床应用标准。
摘要：
大型语言模型（LLMs）在许多领域的应用日益增多。本文评估了三种 LLM 聊天机器人（ChatGPT-3.5、ChatGPT-4 和 Bard）——以其当前公开可用的形式——利用自发语音记录中的文本输入来识别阿尔茨海默病性痴呆（AD）和认知正常（CN）个体的能力。采用零样本学习方法，在两个独立查询级别进行，第二个查询（思维链提示）比第一个引发更详细的回答。每个 LLM 聊天机器人的性能根据生成的预测，以准确率、灵敏度、特异性、精确率和 F1 分数进行评估。LLM 聊天机器人生成了三类结果（“AD”、“CN”或“不确定”）。在积极识别 AD 时，Bard 产生了最高的真阳性率（89% 召回率）和最高的 F1 分数（71%），但倾向于将 CN 误识别为 AD，且置信度较高（“不确定”率低）；在积极识别 CN 时，GPT-4 产生了最高的真阴性率（56%）和最高的 F1 分数（62%），采取了外交式立场（“不确定”率中等）。总体而言，三种 LLM 聊天机器人识别 AD 与 CN 的能力超过了随机水平，但目前尚不能满足临床应用要求。
英文摘要：
Large language models (LLMs) find increasing applications in many fields. Here, three LLM chatbots (ChatGPT-3.5, ChatGPT-4 and Bard) are assessed - in their current form, as publicly available - for their ability to recognize Alzheimer's Dementia (AD) and Cognitively Normal (CN) individuals using textual input derived from spontaneous speech recordings. Zero-shot learning approach is used at two levels of independent queries, with the second query (chain-of-thought prompting) eliciting more detailed than the first. Each LLM chatbot's performance is evaluated on the prediction generated in terms of accuracy, sensitivity, specificity, precision and F1 score. LLM chatbots generated three-class outcome ("AD", "CN", or "Unsure"). When positively identifying AD, Bard produced highest true-positives (89% recall) and highest F1 score (71%), but tended to misidentify CN as AD, with high confidence (low "Unsure" rates); for positively identifying CN, GPT-4 resulted in the highest true-negatives at 56% and highest F1 score (62%), adopting a diplomatic stance (moderate "Unsure" rates). Overall, three LLM chatbots identify AD vs CN surpassing chance-levels but do not currently satisfy clinical application.
ADRD-Bench: A Preliminary LLM Benchmark for Alzheimer's Disease and Related Dementias
Electrical Engineering, University of Notre Dame（诺丁汉大学电气工程系）；Computer Science and Engineering, University of Notre Dame（诺丁汉大学计算机科学与工程系）；School of Medicine, Indiana University（印第安纳大学医学院）
Mapping Caregiver Needs to AI Chatbot Design: Strengths and Gaps in Mental Health Support for Alzheimer's and Dementia Caregivers
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）
RWTH Aachen University（亚琛工业大学）；University of Amsterdam（阿姆斯特丹大学）
In this paper, we combined linguistic complexity and (dis)fluency features with pretrained language models for the task of Alzheimer's disease detection of the 2021 ADReSSo (Alzheimer's Dementia Recognition through Spontaneous Speech) challenge. An accuracy of 83.1% was achieved on the test set, which amounts to an improvement of 4.23% over the baseline model. Our best-performing model that integrated component models using a stacking ensemble technique performed equally well on cross-validation and test data, indicating that it is robust against overfitting.
Usher Institute, Edinburgh Medical School, The University of Edinburgh（爱丁堡大学爱丁堡医学院厄舍研究所）；Carnegie Mellon University（卡内基梅隆大学）；Information Technologies Institute, CERTH（希腊研究与技术基金会信息技术研究所）
This Signal Processing Grand Challenge (SPGC) targets a difficult automatic prediction problem of societal and medical relevance, namely, the detection of Alzheimer's Dementia (AD). Participants were invited to employ signal processing and machine learning methods to create predictive models based on spontaneous speech data. The Challenge has been designed to assess the extent to which predictive models built based on speech in one language (English) generalise to another language (Greek). To the best of our knowledge no work has investigated acoustic features of the speech signal in multiling…
University of Washington（华盛顿大学）；University of Minnesota（明尼苏达大学）
In recent years there has been a burgeoning interest in the use of computational methods to distinguish between elicited speech samples produced by patients with dementia, and those from healthy controls. The difference between perplexity estimates from two neural language models (LMs) - one trained on transcripts of speech produced by healthy participants and the other trained on transcripts from patients with dementia - as a single feature for diagnostic classification of unseen transcripts has been shown to produce state-of-the-art performance. However, little is known about why this approa…
Alzheimer's disease (AD) is an irreversible and progressive brain disease that can be stopped or slowed down with medical treatment. Language changes serve as a sign that a patient's cognitive functions have been impacted, potentially leading to early diagnosis. In this work, we use NLP techniques to classify and analyze the linguistic characteristics of AD patients using the DementiaBank dataset. We apply three neural models based on CNNs, LSTM-RNNs, and their combination, to distinguish between language samples from AD and control patients. We achieve a new independent benchmark accuracy for…
BITS Pilani, K.K. Birla Goa Campus（Birla技术与科学学院皮拉尼校区 K.K. Birla果阿校区）；IEMN Université de Lille（里尔大学微电子与纳米技术研究所）
Alzheimers disease is a fatal progressive brain disorder that worsens with time. It is high time we have inexpensive and quick clinical diagnostic techniques for early detection and care. In previous studies, various Machine Learning techniques and Pre-trained Deep Learning models have been used in conjunction with the extraction of various acoustic and linguistic features. Our study focuses on three models for the classification task in the ADReSS (The Alzheimers Dementia Recognition through Spontaneous Speech) 2021 Challenge. We use the well-balanced dataset provided by the ADReSS Challenge …
Automatic Speech Recognition (ASR) is a critical component of any fully-automated speech-based dementia detection model. However, despite years of speech recognition research, little is known about the impact of ASR accuracy on dementia detection. In this paper, we experiment with controlled amounts of artificially generated ASR errors and investigate their influence on dementia detection. We find that deletion errors affect detection performance the most, due to their impact on the features of syntactic complexity and discourse representation in speech. We show the trend to be generalisable a…
Collecting and accessing a large amount of medical data is very time-consuming and laborious, not only because it is difficult to find specific patients but also because it is required to resolve the confidentiality of a patient's medical records. On the other hand, there are deep learning models, trained on easily collectible, large scale datasets such as Youtube or Wikipedia, offering useful representations. It could therefore be very advantageous to utilize the features from these pre-trained networks for handling a small amount of data at hand. In this work, we exploit various multi-modal …
Alzheimer's Dementia (AD) is an incurable, debilitating, and progressive neurodegenerative condition that affects cognitive function. Early diagnosis is important as therapeutics can delay progression and give those diagnosed vital time. Developing models that analyse spontaneous speech could eventually provide an efficient diagnostic modality for earlier diagnosis of AD. The Alzheimer's Dementia Recognition through Spontaneous Speech task offers acoustically pre-processed and balanced datasets for the classification and prediction of AD and associated phenotypes through the modelling of spont…
Queen Mary University of London（伦敦玛丽女王大学）；Jožef Stefan Institute（约热夫·斯特凡研究所）
We present two multimodal fusion-based deep learning models that consume ASR transcribed speech and acoustic data simultaneously to classify whether a speaker in a structured diagnostic task has Alzheimer's Disease and to what degree, evaluating the ADReSSo challenge 2021 data. Our best model, a BiLSTM with highway layers using words, word probabilities, disfluency features, pause information, and a variety of acoustic features, achieves an accuracy of 84% and RSME error prediction of 4.26 on MMSE cognitive scores. While predicting cognitive decline is more challenging, our models show improve…
Swin-BERT: A Feature Fusion System designed for Speech-based Alzheimer's Dementia Detection
College of Artificial Intelligence, Dalian Maritime University（大连海事大学人工智能学院）；University of Sheffield（谢菲尔德大学）；College of Information Science and Technology, Dalian Maritime University（大连海事大学信息科学技术学院）
Introduction: Alzheimer's disease is a type of dementia in which early diagnosis plays a major rule in the quality of treatment. Among new works in the diagnosis of Alzheimer's disease, there are many of them analyzing the voice stream acoustically, syntactically or both. The mostly used tools to perform these analysis usually include machine learning techniques. Objective: Designing an automatic machine learning based diagnosis system will help in the procedure of early detection. Also, systems, using noninvasive data are preferable. Methods: We used are classification system based on spoken …
Alzheimer's Disease (AD) is the most common type of dementia, comprising 60-80% of cases. There were an estimated 5.8 million Americans living with Alzheimer's dementia in 2019, and this number will almost double every 20 years. The total lifetime cost of care for someone with dementia is estimated to be $350,174 in 2018, 70% of which is associated with family-provided care. Most family caregivers face emotional, financial and physical difficulties. As a medium to relieve this burden, online communities in social media websites such as Twitter, Reddit, and Yahoo! Answers provide potential venu…
Benchmarking Foundation Speech and Language Models for Alzheimer's Disease and Related Dementia Detection from Spontaneous Speech
Georgia Institute of Technology（佐治亚理工学院）；Shandong Mental Health Center（山东精神卫生中心）；University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德河谷分校）；Kennesaw State University（肯尼索州立大学）
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；Academia Sinica（中央研究院）；Toronto Rehabilitation Institute（多伦多康复研究所）
Machine learning has shown promise for automatic detection of Alzheimer's disease (AD) through speech; however, efforts are hampered by a scarcity of data, especially in languages other than English. We propose a method to learn a correspondence between independently engineered lexicosyntactic features in two languages, using a large parallel corpus of out-of-domain movie dialogue data. We apply it to dementia detection in Mandarin Chinese, and demonstrate that our method outperforms both unilingual and machine translation-based baselines. This appears to be the first study that transfers feat…
Alzheimer's disease (AD) constitutes a neurodegenerative disease with serious consequences to peoples' everyday lives, if it is not diagnosed early since there is no available cure. Alzheimer's is the most common cause of dementia, which constitutes a general term for loss of memory. Due to the fact that dementia affects speech, existing research initiatives focus on detecting dementia from spontaneous speech. However, little work has been done regarding the conversion of speech data to Log-Mel spectrograms and Mel-frequency cepstral coefficients (MFCCs) and the usage of pretrained models. Con…
Alzheimer's disease (AD) is an irreversible brain disease that can dramatically reduce quality of life, most commonly manifesting in older adults and eventually leading to the need for full-time care. Early detection is fundamental to slowing its progression; however, diagnosis can be expensive, time-consuming, and invasive. In this work we develop a neural model based on a CNN-LSTM architecture that learns to detect AD and related dementias using targeted and implicitly-learned features from conversational transcripts. Our approach establishes the new state of the art on the DementiaBank data…
