BrokenBind: 超越数据集边界的通用模态探索
BrokenBind: Universal Modality Exploration beyond Dataset Boundaries
AI总结：
BrokenBind提出了一种超越数据集边界的通用模态探索方法，通过结合不同数据集的模态实现灵活且通用的多模态学习。
摘要：
多模态学习通过结合各种模态来提供对现实世界问题的全面理解。一种常见策略是直接在特定的联合嵌入空间中绑定不同的模态。然而，现有方法的能力受限于给定数据集中提供的模态，因此在下游任务中泛化到未呈现的模态时会受到偏见。因此，由于这种不灵活，先前方法的可行性受到获取多模态数据集成本的严重阻碍。在本文中，我们引入了BrokenBind，其重点是绑定来自不同数据集的模态。为了实现这一目标，BrokenBind同时利用包含所关注模态的多个数据集和一个共享模态。尽管这两个数据集由于分布不匹配而不对应，我们仍能捕捉它们之间的关系，生成伪嵌入以填补所关注模态的缺失部分，从而实现灵活且通用的多模态学习。在我们的框架下，任何两种模态都可以被绑定在一起，不受数据集限制，以实现通用模态探索。进一步，为了揭示我们方法的能力，我们研究了加强场景，其中需要多个数据集进行模态绑定，并展示了BrokenBind在低数据环境中的有效性。通过广泛的评估，我们仔细证明了BrokenBind相比知名多模态基线方法的优越性。
英文摘要：
Multi-modal learning combines various modalities to provide a comprehensive understanding of real-world problems. A common strategy is to directly bind different modalities together in a specific joint embedding space. However, the capability of existing methods is restricted within the modalities presented in the given dataset, thus they are biased when generalizing to unpresented modalities in downstream tasks. As a result, due to such inflexibility, the viability of previous methods is seriously hindered by the cost of acquiring multi-modal datasets. In this paper, we introduce BrokenBind, which focuses on binding modalities that are presented from different datasets. To achieve this, BrokenBind simultaneously leverages multiple datasets containing the modalities of interest and one shared modality. Though the two datasets do not correspond to each other due to distribution mismatch, we can capture their relationship to generate pseudo embeddings to fill in the missing modalities of interest, enabling flexible and generalized multi-modal learning. Under our framework, any two modalities can be bound together, free from the dataset limitation, to achieve universal modality exploration. Further, to reveal the capability of our method, we study intensified scenarios where more than two datasets are needed for modality binding and show the effectiveness of BrokenBind in low-data regimes. Through extensive evaluation, we carefully justify the superiority of BrokenBind compared to well-known multi-modal baseline methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
