使用卷积神经网络评估精神分裂症神经运动协调性的多模态方法
Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks
AI总结：
本研究提出一种多模态卷积神经网络，利用视频和音频数据分析精神分裂症患者的语音发音协调模式，发现声道变量结合面部动作单元优于MFCCs，检测F1分数较FVTC基线提升约18%。
摘要：
本研究调查了表现出强阳性症状（如幻觉和妄想）的精神分裂症受试者的语音发音协调性，使用了两种不同的声道延迟相关方法。我们表明，具有强阳性症状且病情明显的精神分裂症受试者在面部和语音手势中表现出比健康受试者更复杂的发音协调模式。这种语音协调模式的区别被用来训练一个多模态卷积神经网络（CNN），该网络在语音过程中使用视频和音频数据来区分具有强阳性症状的精神分裂症患者与健康受试者。我们还表明，与发音部位和声门源相对应的声道变量（TVs）在所提出的多模态网络中与面部动作单元（FAUs）融合时，优于Mel频率倒谱系数（MFCCs）。对于我们收集的临床数据集，我们表现最好的多模态网络在检测精神分裂症的平均F1分数上，相对于融合FAUs和MFCCs的完整声道协调（FVTC）基线方法提高了约18%。
英文摘要：
This study investigates the speech articulatory coordination in schizophrenia subjects exhibiting strong positive symptoms (e.g. hallucinations and delusions), using two distinct channel-delay correlation methods. We show that the schizophrenic subjects with strong positive symptoms and who are markedly ill pose complex articulatory coordination pattern in facial and speech gestures than what is observed in healthy subjects. This distinction in speech coordination pattern is used to train a multimodal convolutional neural network (CNN) which uses video and audio data during speech to distinguish schizophrenic patients with strong positive symptoms from healthy subjects. We also show that the vocal tract variables (TVs) which correspond to place of articulation and glottal source outperform the Mel-frequency Cepstral Coefficients (MFCCs) when fused with Facial Action Units (FAUs) in the proposed multimodal network. For the clinical dataset we collected, our best performing multimodal network improves the mean F1 score for detecting schizophrenia by around 18% with respect to the full vocal tract coordination (FVTC) baseline method implemented with fusing FAUs and MFCCs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
