AdCare-VLM：面向医疗视频理解的统一预对齐潜在表示研究
AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding
AI总结：
针对慢性病患者服药依从性监测需求，提出基于LLaVA的专用多模态大模型AdCare-VLM，通过预对齐统一视觉潜在空间提升相关VQA性能，在肺结核服药监测任务上效果优于同类基线模型。
摘要：
糖尿病、高血压、哮喘、艾滋病、癫痫和肺结核等慢性病需要患者严格遵医嘱服药，才能避免病情恶化、控制症状并降低死亡率。但患者行为、照护者支持情况、高昂医疗成本、医疗基础设施不足等因素常导致服药依从性不佳。本文提出AdCare-VLM，这是一款基于LLaVA的专用多模态视觉大语言模型（LVLM），通过引入经预对齐的统一视觉潜在空间，支持基于患者视频的服药依从性视觉问答（VQA）任务。研究使用包含806条经临床专家标注的肺结核（TB）服药监测私有视频数据集微调模型，用于依从性模式检测，同时构建了覆盖依从性正向、负向、模糊案例的细粒度医疗依从性VQA数据集LLM-TB-VQA。该方法可捕捉患者面部清晰可见度、药物、饮水、吞咽动作等视觉特征与字幕中对应医学概念的关联，实现对齐的视觉-语言表示融合，提升多模态交互效果。实验结果显示，该方法在预训练、常规微调、低秩适配（LoRA）配置下，性能均优于采用参数高效微调（PEFT）的VLM模型（包括LLaVA-V1.5、Chat-UniVi），绝对提升幅度达3.1%至3.54%。充分的消融实验与注意力图可视化验证了方法有效性，提升了模型可解释性。
英文摘要：
Chronic diseases, including diabetes, hypertension, asthma, HIV-AIDS, epilepsy, and tuberculosis, necessitate rigorous adherence to medication to avert disease progression, manage symptoms, and decrease mortality rates. Adherence is frequently undermined by factors including patient behavior, caregiver support, elevated medical costs, and insufficient healthcare infrastructure. We propose AdCare-VLM, a specialized LLaVA-based multimodal large vision language model (LVLM) by introducing a unified visual latent space with pre-alignment to facilitate visual question answering (VQA) concerning medication adherence through patient videos. We employ a private dataset comprising 806 custom-annotated tuberculosis (TB) medication monitoring videos, which have been labeled by clinical experts, to fine-tune the model for adherence pattern detection. We present LLM-TB-VQA, a detailed medical adherence VQA dataset that encompasses positive, negative, and ambiguous adherence cases. Our method identifies correlations between visual features, such as the clear visibility of the patient's face, medication, water intake, and the act of ingestion, and their associated medical concepts in captions. This facilitates the integration of aligned visual-linguistic representations and improves multimodal interactions. Experimental results indicate that our method surpasses parameter-efficient fine-tuning (PEFT) enabled VLM models, such as LLaVA-V1.5 and Chat-UniVi, with absolute improvements ranging from 3.1% to 3.54% across pre-trained, regular, and low-rank adaptation (LoRA) configurations. Comprehensive ablation studies and attention map visualizations substantiate our approach, enhancing interpretability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
