arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.05561cs.CV
收藏

利用多示例学习构建多模态表示

Using Multiple Instance Learning to Build Multimodal Representations

MIT(麻省理工学院);Harvard Medical School(哈佛医学院)

Peiqi Wang, William M. Wells, Seth Berkowitz, Steven Horng, Polina Golland

AI总结:

本研究建立多模态表示学习与多示例学习的关联,提出构建置换不变评分函数的通用框架,并据此推导新型对比学习方法,在多项下游任务中取得最优结果。

摘要:

图像-文本多模态表示学习可跨模态对齐数据,支撑图像分类、视觉定位、跨模态检索等重要医学应用。本研究建立了多模态表示学习与多示例学习(Multiple Instance Learning)的关联,基于该关联提出构建置换不变评分函数的通用框架,现有多种多模态表示学习方法均为其特例。此外,我们借助该框架推导出一种新型对比学习方法,实验证明该方法在多项下游任务上取得了state-of-the-art(最优)结果。

英文摘要:

Image-text multimodal representation learning aligns data across modalities and enables important medical applications, e.g., image classification, visual grounding, and cross-modal retrieval. In this work, we establish a connection between multimodal representation learning and multiple instance learning. Based on this connection, we propose a generic framework for constructing permutation-invariant score functions with many existing multimodal representation learning approaches as special cases. Furthermore, we use the framework to derive a novel contrastive learning approach and demonstrate that our method achieves state-of-the-art results in several downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑