MedicalNarratives: 通过局部化叙述连接医学视觉与语言
MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives
AI总结：
MedicalNarratives通过局部化鼠标轨迹连接医学视觉与语言，训练出的GenMedClip在12个医学领域均优于现有最佳模型。
摘要：
多模态模型对数据需求很高。虽然自然图像数据集丰富，但医学图像数据集却无法承担同样的奢侈。为了在大规模上进行医学图像的表示学习，我们转向YouTube，一个拥有大量开源医学教学视频的平台。我们整理了MedicalNarratives数据集，包含470万张医学图像-文本对，其中100万张样本包含密集标注，形式为空间轨迹（和边界框），以及11.8万部围绕轨迹事件（与文本对齐）的视频，使空间时间定位超越单帧。类似于think-aloud研究，其中讲师在相关图像区域上悬停鼠标时说话，MedicalNarratives中的100万张图像包含局部化鼠标轨迹，创建了文本与像素之间的空间和时间关联。为了评估MedicalNarratives的实用性，我们使用我们的数据集跨越12个医学领域训练GenMedClip，使用类似于CLIP的目标函数。GenMedClip在新构建的医学影像基准上所有12个领域均优于先前的最先进模型。[数据]
英文摘要：
Multi-modal models are data hungry. While datasets with natural images are abundant, medical image datasets can not afford the same luxury. To enable representation learning for medical images at scale, we turn to YouTube, a platform with a large reservoir of open-source medical pedagogical videos. We curate MedicalNarratives, a dataset 4.7M medical image-text pairs, with 1M samples containing dense annotations in the form of spatial traces (and bounding boxes), and 118K videos centered on the trace event (with aligned text), enabling spatiotemporal grounding beyond single frames. Similar to $\textit{think-aloud}$ studies where instructors speak while hovering their mouse cursor movements over relevant image regions, 1M images in MedicalNarratives contains localized mouse traces in image pixels, creating a spatial and temporal association between the text and pixels. To evaluate the utility of MedicalNarratives, we train GenMedClip with a CLIP-like objective using our dataset spanning 12 medical domains. GenMedClip outperforms previous state-of-the-art models on all 12 domains on a newly constructed medical imaging benchmark. $\href{https://huggingface.co/datasets/wisdomik/MedicalNarratives}{[Data]}$
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
