多模态特权知识蒸馏在两种基于视觉Transformer的诊断应用中的有效性研究
On the effectiveness of multimodal privileged knowledge distillation in two vision transformer based diagnostic applications
AI总结：
本文提出多模态特权知识蒸馏（MMPKD）策略，利用仅在训练时可用的额外模态指导单模态视觉Transformer模型，在胸部X光和乳腺X光诊断中提升了零样本ROI定位能力，但该效果不跨领域泛化。
摘要：
在临床实践中部署深度学习模型通常需要利用图像、文本和结构化数据等多种数据模态，以实现稳健且可信的决策。然而，并非所有模态在推理时总是可用的。在本研究中，我们提出了多模态特权知识蒸馏（MMPKD），这是一种仅利用训练期间可用的额外模态来指导单模态视觉模型的训练策略。具体而言，我们使用基于文本的教师模型处理胸部X光片（MIMIC-CXR），并使用基于表格元数据的教师模型处理乳腺X光检查（CBIS-DDSM），将知识蒸馏到视觉Transformer学生模型中。我们表明，MMPKD能够提升所得注意力图在输入图像中定位ROI的零样本能力，而这种效果并不能跨领域泛化，这与先前研究的建议相反。
英文摘要：
Deploying deep learning models in clinical practice often requires leveraging multiple data modalities, such as images, text, and structured data, to achieve robust and trustworthy decisions. However, not all modalities are always available at inference time. In this work, we propose multimodal privileged knowledge distillation (MMPKD), a training strategy that utilizes additional modalities available solely during training to guide a unimodal vision model. Specifically, we used a text-based teacher model for chest radiographs (MIMIC-CXR) and a tabular metadata-based teacher model for mammography (CBIS-DDSM) to distill knowledge into a vision transformer student model. We show that MMPKD can improve the resulting attention maps' zero-shot capabilities of localizing ROI in input images, while this effect does not generalize across domains, as contrarily suggested by prior research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
