Early identification of stroke symptoms is essential for enabling timely intervention and improving patient outcomes, particularly in prehospital settings. This study presents a fast, non-invasive multimodal deep learning framework for automatic binary stroke screening based on data collected during the F.A.S.T. assessment. The proposed approach integrates complementary information from facial expressions, speech signals, and upper-body movements to enhance diagnostic robustness. Facial dynamics are represented using landmark based features and modeled with a Transformer architecture to capture temporal dependencies. Speech signals are converted into mel spectrograms and processed using an Audio Spectrogram Transformer, while upper-body pose sequences are analyzed with an MLP-Mixer network to model spatiotemporal motion patterns. The extracted modality specific representations are combined through an attention-based fusion mechanism to effectively learn cross modal interactions. Experiments conducted on a self-collected dataset of 222 videos from 37 subjects demonstrate that the proposed multimodal model consistently outperforms unimodal baselines, achieving 95.83% accuracy and a 96.00% F1-score. The model attains a strong balance between sensitivity and specificity and successfully detects all stroke cases in the test set. These results highlight the potential of multimodal learning and transfer learning for early stroke screening, while emphasizing the need for larger, clinically representative datasets to support reliable real-world deployment.
ISLES'24 -- A Real-World Longitudinal Multimodal Stroke Dataset
Technical University of Munich(慕尼黑工业大学);University of Zurich(苏黎世大学);University Hospital of Zurich(苏黎世大学医院);Zurich University of Applied Sciences(苏黎世应用科学大学);icometrix(icometrix公司);University of Girona(赫罗纳大学);Inselspital(伯尔尼医院);University Hospital Bern(伯尔尼大学医院);University of Bern(伯尔尼大学);Eindhoven University of Technology(埃因霍温理工大学);Imperial College London(帝国理工学院)
STROKEVISION-BENCH: A Multimodal Video And 2D Pose Benchmark For Tracking Stroke Recovery
Center for Research in Computer Vision, University of Central Florida(计算机视觉研究中心,佛罗里达中央大学);Mechanical and Aerospace Engineering, University of Central Florida(机械与航空航天工程,佛罗里达中央大学)
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
UMIT TIROL – Private University for Health Sciences and Health Technology(UMIT TIROL – 健康科学与健康技术私立大学);VASCage – Centre on Clinical Stroke Research(VASCage – 临床卒中研究中心);Medical University of Innsbruck(因斯布鲁克医科大学)
ISLES'24: Final Infarct Prediction with Multimodal Imaging and Clinical Data. Where Do We Stand?
Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland.;Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands.;ARTORG Center for Biomedical Research, University of Bern, Bern, Switzerland.;Department of Radiation Oncology, University Hospital Bern, University of Bern.;Support Center of Advanced Neuroimaging (SCAN), University Institute of Diagnostic;University Institute of Diagnostic;Interventional Neuroradiology, University Hospital Bern, Inselspital, University of Bern, Bern, Switzerland.;Department of Neurology, University Hospital of Zurich, Zurich, Switzerland.;University of Zurich, Zurich, Switzerland.;Department of Diagnostic;Interventional Neuroradiology, School of Medicine;Health, TUM Klinikum, Technical University of Munich, Germany.;TranslaTUM, Center for Translational Cancer Research, Technical University of Munich, Germany.;Therapy, School of Medicine;Health, Technical University of Munich, Munich, Germany.;Department of Biomedical Engineering;Physics, Amsterdam UMC location University of Amsterdam, Amsterdam, the Netherlands;Department of Radiology, University of Calgary, Calgary, Canada;Hotchkiss Brain Institute, University of Calgary, Calgary, Canada;Alberta Children’s Hospital Research Institute, University of Calgary, Calgary, Canada;Heuron Co., Ltd., Seoul, South Korea;School of Electrical Engineering, Korea University, Seoul, Korea;University of Alberta, Alberta, Canada;Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom;Lung Institute, Faculty of Medicine, Imperial College London, London, United Kingdom;University of Calgary, Canada;University of Washington, Washington, United States;CREATIS, Universite Claude Bernard Lyon 1, INSA Lyon, UMR CNRS 5220, Inserm U1294, Villeurbanne, France;Department of Neurology, University Hospital Zurich, Zurich, Switzerland
LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
School of Medicine, Yale University(耶鲁大学医学院);School of Computing, Australian National University(澳大利亚国立大学计算机学院);School of Engineering, Imperial College London(伦敦帝国理工学院工程学院);School of Computing, University of Georgia(佐治亚大学计算机学院);Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨秀隆医学学院);National Eye Institute, National Institutes of Health(美国国立卫生研究院眼科研究所);National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算机技术学院)
Multimodal Deep Learning for Stroke Prediction and Detection using Retinal Imaging and Clinical Data
Division of Engineering, New York University Abu Dhabi(纽约大学阿布扎赫尔分校工程系);Institute for Healthier Living Abu Dhabi(阿布扎赫尔健康生活研究所);Eye Institute at Cleveland Clinic Abu Dhabi(阿布扎赫尔克利夫兰医学中心眼科研究所);Canberra Hospital(堪培拉医院)
Towards Clinical Diagnosis: Automated Stroke Lesion Segmentation on Multimodal MR Image Using Convolutional Neural Network
Nankai University(南开大学);Tianjin Huanhu Hospital(天津市环湖医院);the University of Aizu(会津大学);Tianjin Medical University(天津医科大学);College of Electronic Information and Optical Engineering, Nankai University(南开大学电子信息与光学工程学院);School of Computer Science and Engineering, the University of Aizu(会津大学计算机科学与工程学院);School of Basic Medical Sciences, Tianjin Medical University(天津医科大学基础医学院);Department of Medical Imaging, Tianjin Huanhu Hospital(天津市环湖医院医学影像科);Key Laboratory for Cerebral Artery and Neural Degeneration of Tianjin(天津市脑血管及神经变性重点实验室);Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology(天津市光电传感器与传感网络技术重点实验室)
Advancing Stroke Risk Prediction Using a Multi-modal Foundation Model
Signal Processing Institute EPFL University Lausanne(瑞士洛桑联邦理工学院信号处理研究所);Brigham and Women’s Hospital Harvard Medical School(哈佛医学院布里奇沃特医院);Department of Quantitative Biomedicine University of Zurich(苏黎世大学定量生物医学系);ETH AI Center, Department of Computer Science Department of Quantitative Biomedicine University of Zurich(苏黎世大学定量生物医学系)
Dual-Encoder Transformer-Based Multimodal Learning for Ischemic Stroke Lesion Segmentation Using Diffusion MRI
Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University(麻醉学、围术期与疼痛医学系,斯坦福大学);Department of Biomedical Sciences, Seoul National University(生物医学科学系,首尔国立大学);Department of Computer Engineering, National University of Sciences and Technology (NUST)(计算机工程系,国立科学与技术大学(NUST));Department of Computer Science, The University of Alabama(计算机科学系,阿拉巴马大学);Department of Computer Science, Hanyang University(计算机科学系,翰阳大学)
Comparing Baseline and Day-1 Diffusion MRI Using Multimodal Deep Embeddings for Stroke Outcome Prediction
1 Department of Computer Science, University of Geneva, Switzerland 2 Department of Electrical \& Computer Engineering, University of Florida, FL, USA 3 Service of Medical Informatics, University Hospital of Geneva, Switzerland 4 Department of Imaging;Medical Informatics, University of Geneva, Switzerland
MOSCARD -- Causal Reasoning and De-confounding for Multimodal Opportunistic Screening of Cardiovascular Adverse Events
Department of Data Science & Engineering, Arizona State University(数据科学与工程系,亚利桑那州立大学);Department of Cardiovascular Medicine, Mayo Clinic(心血管医学系,梅奥诊所);Department of Radiology, Mayo Clinic(放射科,梅奥诊所);Catholic kwandong university international st. mary’s hospital(韩国天主教 kwandong 大学国际圣玛丽医院)
Structured Visual Evidence Decomposition for Evidence-Grounded Multimodal Screening of Obstructive Sleep Apnea-Hypopnea Syndrome
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science(上海工程技术大学电子与电气工程学院);Tencent Youtu Lab(腾讯云视频实验室);ENT Institute and Department of Otorhinolaryngology, Eye & ENT Hospital of Fudan University(复旦大学耳鼻喉科医院耳鼻喉科研究所);National University of Singapore(新加坡国立大学)
A multimodal large language model for evidence-based autism spectrum disorder screening
Zhejiang Normal University(浙江师范大学);State Grid Corporation of China(国家电网有限公司);Huzhou Normal University(湖州师范学院);Zhejiang University of Technology(浙江工业大学)
Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
Meta Superintelligence Labs(Meta超级智能实验室);University of California, San Diego(加州大学圣迭戈分校);Worcester Polytechnic Institute(伍斯特理工学院);Northwestern University(西北大学)
Seeing Is Not Screening: Multimodal Hidden Instruction Attacks on Agent Skill Scanners
Nanyang Technological University, Singapore(南洋理工大学,新加坡);Chongqing University, China(重庆大学);Northeastern University, China(东北大学);University of Chinese Academy of Sciences, China(中国科学院大学);Beihang University, China(北航大学)
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute(伦斯勒理工学院);Wake Forest University School of Medicine(维克森林大学医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening
Lausanne University Hospital(洛桑大学医院);Faculty of Biology and Medicine, University of Lausanne(洛桑大学生物与医学学院);VNU University of Engineering and Technology(越南国立大学工程技术大学);VinUni-Illinois Smart Health Center, VinUniversity(文大-伊利诺伊智能健康中心,文大);University of Science, VNU-HCM(胡志明市越南国立大学科学大学);Hanoi Medical University(河内医科大学);National Geriatric Hospital(国家老年病医院);Military Hospital 175(第175军医院);School of Medicine, University of Medicine and Pharmacy at Ho Chi Minh City(胡志明市医药大学医学院);International University, VNU-HCM(胡志明市越南国立大学国际大学);Vietnam National University Ho Chi Minh City(胡志明市越南国立大学)
Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection
Wyant College of Optical Sciences, University of Arizona(亚利桑那大学怀恩特光学科学学院);Beckman Laser Institute & Medical Clinic, University of California Irvine(加州大学尔湾分校贝克曼激光研究所与医学诊所)