使用 LoReTTa 训练具有传递性与交换性的多模态 Transformer
Training Transitive and Commutative Multimodal Transformers with LoReTTa
AI总结:
本文提出 LoReTTa 自监督预训练策略,通过传递性和交换性规则统一因果建模与掩码建模,使多模态 Transformer 能处理推理时缺失或未见的模态组合,并在合成、医学和强化学习数据集上超越 GPT、BERT 和 CLIP 等基线。
摘要:
由于多模态数据集的可用性有限,训练多模态基础模型具有挑战性。虽然许多公开数据集将图像与文本配对,但很少有数据集将图像与音频或文本与音频组合在一起。同时对齐所有三种模态的数据集则更为罕见。医疗保健、基础设施或交通运输等关键领域尤其受到模态缺失的影响。这使得将所有模态整合到一个大型预训练神经网络中变得困难,而该网络本应能够开箱即用或针对不同下游任务进行微调。我们引入 LoReTTa(Linking mOdalities with a tRansitive and commutativE pre-Training sTrAtegy,即通过传递性和交换性预训练策略连接模态)来解决这一研究不足的问题。我们的自监督框架将因果建模和掩码建模与交换性和传递性规则统一起来。这使我们能够在模态内部和模态之间进行转换。因此,我们的预训练模型能够更好地探索真实的底层联合概率分布。给定一个仅包含不相交组合 (A, B) 和 (B, C) 的数据集,LoReTTa 可以通过 A <-> B <-> C 来建模关系 A <-> C。特别是,我们证明了使用 LoReTTa 预训练的 Transformer 在推理时可以处理任意模态混合,包括从未见过的模态对 (A, C) 和三元组 (A, B, C)。我们在合成数据集、医学数据集和强化学习数据集上对我们的方法进行了广泛评估。在不同领域中,我们的通用多模态 Transformer 在涉及缺失模态元组的任务上始终优于 GPT、BERT 和 CLIP 等强基线模型。
英文摘要:
Training multimodal foundation models is challenging due to the limited availability of multimodal datasets. While many public datasets pair images with text, few combine images with audio or text with audio. Even rarer are datasets that align all three modalities at once. Critical domains such as healthcare, infrastructure, or transportation are particularly affected by missing modalities. This makes it difficult to integrate all modalities into a large pre-trained neural network that can be used out-of-the-box or fine-tuned for different downstream tasks. We introduce LoReTTa (Linking mOdalities with a tRansitive and commutativE pre-Training sTrAtegy) to address this understudied problem. Our self-supervised framework unifies causal modeling and masked modeling with the rules of commutativity and transitivity. This allows us to transition within and between modalities. As a result, our pre-trained models are better at exploring the true underlying joint probability distribution. Given a dataset containing only the disjoint combinations (A, B) and (B, C), LoReTTa can model the relation A <-> C with A <-> B <-> C. In particular, we show that a transformer pre-trained with LoReTTa can handle any mixture of modalities at inference time, including the never-seen pair (A, C) and the triplet (A, B, C). We extensively evaluate our approach on a synthetic, medical, and reinforcement learning dataset. Across different domains, our universal multimodal transformer consistently outperforms strong baselines such as GPT, BERT, and CLIP on tasks involving the missing modality tuple.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。