向邻居借用宝藏：面向模态缺失与数据稀缺多模态学习的上下文学习
Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcity
AI总结：
针对多模态学习中模态缺失与数据稀缺的双重挑战，提出基于检索增强的上下文学习框架，利用全模态数据提升样本效率，在低数据场景下显著增强分类性能并缩小模态间性能差距。
摘要：
存在模态缺失的多模态机器学习是医疗保健等各种应用中日益相关的挑战。本文将当前关于模态缺失的研究扩展到低数据场景，即下游任务同时存在模态缺失和样本量有限的问题。这种问题设置极具挑战性且非常实用，因为获取全模态数据和足够的带标注训练样本通常成本高昂。我们提出使用检索增强的上下文学习，通过释放 transformer 的上下文学习能力来解决这两个关键问题。现有方法主要属于参数化范式且通常需要足够的训练样本，与此不同，我们的工作利用了可用全模态数据的价值，为解决该挑战提供了新视角。所提出的数据依赖框架展现出更高的样本效率，并且经实证证明，在各种多模态学习任务的低数据场景下，能够增强分类模型在全模态和缺失模态数据上的性能。当仅有 1% 的训练数据可用时，我们提出的方法在各种数据集和缺失状态下，相比近期强大的 baseline 平均提升了 6.1%。值得注意的是，与 baseline 相比，我们的方法还缩小了全模态与缺失模态数据之间的性能差距。
英文摘要：
Multimodal machine learning with missing modalities is an increasingly relevant challenge arising in various applications such as healthcare. This paper extends the current research into missing modalities to the low-data regime, i.e., a downstream task has both missing modalities and limited sample size issues. This problem setting is particularly challenging and also practical as it is often expensive to get full-modality data and sufficient annotated training samples. We propose to use retrieval-augmented in-context learning to address these two crucial issues by unleashing the potential of a transformer's in-context learning ability. Diverging from existing methods, which primarily belong to the parametric paradigm and often require sufficient training samples, our work exploits the value of the available full-modality data, offering a novel perspective on resolving the challenge. The proposed data-dependent framework exhibits a higher degree of sample efficiency and is empirically demonstrated to enhance the classification model's performance on both full- and missing-modality data in the low-data regime across various multimodal learning tasks. When only 1% of the training data are available, our proposed method demonstrates an average improvement of 6.1% over a recent strong baseline across various datasets and missing states. Notably, our method also reduces the performance gap between full-modality and missing-modality data compared with the baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
