通过信息瓶颈对齐多模态表示
Aligning Multimodal Representations through an Information Bottleneck
AI总结：
本文从信息瓶颈原理出发，指出对比损失因未去除模态特定信息而难以对齐多模态表示，并提出一种基于变分近似的正则化项以提升表示对齐，在受控实验和实际应用中验证了其有效性。
摘要：
对比损失已被广泛用作多模态表示学习的工具。然而，经验观察表明，使用对比损失并不能有效学习对齐的表示空间。在本文中，我们认为这一现象是由表示空间中存在模态特定信息所导致的。尽管一些最广泛使用的对比损失最大化了两种模态表示之间的互信息，但它们并非旨在去除模态特定信息。我们通过信息瓶颈原理的视角对该问题进行了理论描述。我们还通过受控实验设置，实证分析了不同超参数如何影响这一现象的出现。最后，我们在损失函数中提出了一个正则化项，该项通过变分近似推导得出，旨在提高表示对齐程度。我们在一组受控实验和实际应用中分析了加入该正则化项的优势。
英文摘要：
Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn an aligned representation space. In this paper, we argue that this phenomenon is caused by the presence of modality-specific information in the representation space. Although some of the most widely used contrastive losses maximize the mutual information between representations of both modalities, they are not designed to remove the modality-specific information. We give a theoretical description of this problem through the lens of the Information Bottleneck Principle. We also empirically analyze how different hyperparameters affect the emergence of this phenomenon in a controlled experimental setup. Finally, we propose a regularization term in the loss function that is derived by means of a variational approximation and aims to increase the representational alignment. We analyze in a set of controlled experiments and real-world applications the advantages of including this regularization term.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
