ASR无关的多模态谱时建模用于早期痴呆检测
ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection
AI总结：
提出一种不依赖语音识别的框架，直接从梅尔频谱图中提取谱时位移场作为数字生物标志物，结合CNN-ConvGRU声学嵌入和交叉注意力融合，在三种语言语料库上验证了多模态融合的语料依赖性。
摘要：
语音调用了与日常生活工具性活动（IADL）相同的执行、注意和工作记忆过程，为认知评估提供了非侵入性代理。然而，大多数基于语音的痴呆检测系统依赖于转录，丢弃了录音内的时序结构，并在具有已知录音伪影的单一英语语料库上验证。我们提出了一种ASR无关的框架，直接对梅尔频谱图进行操作。我们的关键贡献是从连续的频谱图帧中提取谱时位移场，捕捉变化的频谱能量模式作为认知衰退的数字生物标志物。这些特征通过学习的交叉注意力机制与CNN-ConvGRU声学嵌入融合，并使用带有可学习查询池化的Transformer编码器进行聚合。复合时间损失强制跨片段的平滑性和对比一致性。我们在英语DementiaBank、斯洛伐克EWA-DB和西班牙语Ivanova语料库上训练独立模型，使用挑战IADL相关认知域的临床诱发协议。斯洛伐克模型达到83.9%的准确率，西班牙语模型达到[原文未给出]，而英语基线为53.2%，确认了已知伪影。跨语言消融研究揭示了不同的融合机制：移除交叉注意力使西班牙语性能降至53.7%，低于单模态模型，而斯洛伐克音频编码器单独表现优于完整模型（93.7% vs 83.9%），所有英语配置仍接近随机水平。因此，多模态融合的价值依赖于语料库：当信号分布在多个模态时必不可少，当某一模态占主导时适得其反，当无信号时无关。辅助时间损失收敛到语言不变的值，表明跨语言架构稳定性。
英文摘要：
Speech recruits the same executive, attentional, and working memory processes underlying instrumental activities of daily living, or IADLs, providing a non-invasive proxy for cognitive assessment. Yet most speech-based dementia detection systems depend on transcription, discard within-recording temporal structure, and are validated on a single English corpus with known recording artifacts. We propose an ASR-agnostic framework operating directly on Mel spectrograms. Our key contribution is extracting spectrotemporal displacement fields from consecutive spectrogram frames, capturing shifting spectral energy patterns as digital biomarkers of cognitive decline. These features are fused with CNN-ConvGRU acoustic embeddings via a learned cross-attention mechanism and aggregated using a Transformer encoder with learnable query pooling. A composite temporal loss enforces smoothness and contrastive coherence across segments. We train independent models on English DementiaBank, Slovak EWA-DB, and Spanish Ivanova corpora, using clinical elicitation protocols taxing IADL-relevant cognitive domains. The Slovak model achieves 83.9% accuracy, and Spanish achieves, while the English baseline yields 53.2%, confirming known artifacts. Cross-lingual ablation studies reveal distinct fusion regimes: removing cross-attention collapses Spanish performance to 53.7%, below unimodal models, while the Slovak audio encoder alone outperforms the full model, 93.7% vs. 83.9%, and all English configurations remain near chance. Thus, multimodal fusion's value is corpus-dependent: essential when signal is distributed across modalities, counterproductive when one dominates, and irrelevant when no signal exists. Auxiliary temporal losses converge to language-invariant values, indicating cross-lingual architectural stability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
