arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20906cs.LGcs.CL
收藏

面向认知障碍检测的任务中心型多模态学习

Task-Centric Multimodal Learning for Cognitive Impairment Detection

University of Massachusetts Boston(马萨诸塞大学波士顿分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)

Sankalp Vaish, Rishank Singh, Sumukh Veeramalla, Caroline Summerour, John A. Batsis, Xiaohui Liang

AI总结:

本研究提出基于Amazon Alexa交互的多模态筛查框架,通过分析视频、音频、文本信号,发现多模态融合模型可稳定区分轻度认知障碍与健康衰老,优于传统单模态模型。

摘要:

痴呆症影响全球数百万老年人,但在临床环境之外的早期检测仍存在困难。传统模型主要依赖基于文本或语音的应答,尽管人类表达本质上是多模态的,却仅能捕捉人类认知的有限视角。本研究提出一种非接触式多模态筛查框架,利用与Amazon Alexa进行结构化语音交互过程中采集的视频、音频和文本信号。通过整合这些模态,我们的方法能捕捉更全面的人类行为表征,涵盖语音模式、语音特征、面部表情和语言结构,这或可实现对认知衰退的更准确检测。不同参与者完成三项会话任务,旨在探究自发语音、理解与回忆能力。我们设定三个目标:(1)评估三项Alexa任务中哪项能提供最具信息量的认知信号;(2)比较单模态模型的有效性;(3)评估多模态融合模型的优势。任务级分析显示,视频模型在指令朗读任务上取得最佳性能,准确率为0.65(AUC为0.69),而音频模型在问答任务上表现最佳,准确率达0.67(AUC为0.73)。相比之下,多模态模型在各项任务中均展现出稳定性能,在指令朗读任务上准确率为0.76(AUC为0.75),在问答任务上准确率为0.76(AUC为0.73)。这些发现表明,通过自然交互采集的多模态行为线索可可靠区分轻度认知障碍(MCI)与健康衰老,凸显了传统诊断范式的局限性,支持向多模态、真实世界认知评估框架转变。

英文摘要:

Dementia affects millions of older adults worldwide, yet early detection remains difficult outside clinical settings. Traditional models primarily rely on text or voice-based responses, capturing only a limited view of human cognition despite the inherently multimodal nature of human expression. This work proposes a contactless, multimodal screening framework that leverages video, audio, and text signals collected during structured voice-based interactions with Amazon Alexa. By incorporating these modalities, our approach captures a more holistic representation of human behavior encompassing speech patterns, vocal characteristics, facial expressions, and linguistic structure, which may enable more accurate detection of cognitive decline. Different participants completed three conversational tasks designed to probe spontaneous speech, comprehension, and recall. We pursued three objectives: (1) evaluate which of the three Alexa tasks provides the most informative cognitive signals, (2) compare the effectiveness of unimodal models, and (3) assess the benefits of multimodal fusion models. Task-level analysis shows that the video model achieves its best performance on Command-Reading, with an accuracy of 0.65 (AUC 0.69), while the audio model performs best on Question-Answering, reaching an accuracy of 0.67 (AUC 0.73). In contrast, the multimodal model demonstrates consistently stable performance across tasks, achieving an accuracy of 0.76 (AUC 0.75) on Command-Reading and 0.76 (AUC 0.73) on Question-Answering. These findings demonstrate that multimodal behavioral cues captured through naturalistic interactions can reliably distinguish MCI from healthy aging, highlighting the limitations of traditional diagnostic paradigms and supporting the shift toward multimodal, real-world cognitive assessment frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑