arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2206.05912cs.CV
收藏

INDIGO:用于领域泛化的内在多峰性

INDIGO: Intrinsic Multimodality for Domain Generalization

Adobe Media and Data Science Research Lab(奥多比媒体与数据科学研究实验室);INRIA(法国国家信息与自动化研究所);Universite Grenoble Alpes(格勒诺布尔阿尔卑斯大学);Indian Institute of Technology, Hyderabad(印度理工学院海得拉巴分校)

Puneet Mangla, Shivam Chandhok, Milan Aggarwal, Vineeth N Balasubramanian, Balaji Krishnamurthy

摘要:

对于在看不见的领域下进行泛化的模型(也称为领域泛化),学习与领域无关的特征表示并捕获构成对象类别的底层语义至关重要。弱监督视觉语言模型从廉价的弱监督噪声文本注释中学习整体表示的最新进展已经通过捕获在不同领域下概括的对象特征来显示其语义理解能力。然而,当涉及多个源域时,为数据集中的每个图像策展文本注释的成本可能会增加几倍,具体取决于它们的数量。这使得这个过程变得繁琐和不可行,阻碍了我们直接使用这些有监督的视觉语言方法来实现对未知领域的最佳泛化。基于此,我们研究了如何以“内在”的方式利用来自现有预训练多模态网络的多模态信息,使系统在看不见的领域下泛化。为此,我们提出了领域泛化(INDIGO)的内在多模态,这是一种简单而优雅的方式,可以利用这些预先训练的多模态网络中存在的内在模态以及视觉模态来增强测试时对未知领域的泛化。我们在几个领域泛化设置(ClosedDG,OpenDG和Limited sources)上进行了实验,并在看不见的领域上展示了最先进的泛化性能。此外,我们提供了一个全面的分析,以发展对靛蓝的全面了解。

英文摘要:

For models to generalize under unseen domains (a.k.a domain generalization), it is crucial to learn feature representations that are domain-agnostic and capture the underlying semantics that makes up an object category. Recent advances towards weakly supervised vision-language models that learn holistic representations from cheap weakly supervised noisy text annotations have shown their ability on semantic understanding by capturing object characteristics that generalize under different domains. However, when multiple source domains are involved, the cost of curating textual annotations for every image in the dataset can blow up several times, depending on their number. This makes the process tedious and infeasible, hindering us from directly using these supervised vision-language approaches to achieve the best generalization on an unseen domain. Motivated from this, we study how multimodal information from existing pre-trained multimodal networks can be leveraged in an "intrinsic" way to make systems generalize under unseen domains. To this end, we propose IntriNsic multimodality for DomaIn GeneralizatiOn (INDIGO), a simple and elegant way of leveraging the intrinsic modality present in these pre-trained multimodal networks along with the visual modality to enhance generalization to unseen domains at test-time. We experiment on several Domain Generalization settings (ClosedDG, OpenDG, and Limited sources) and show state-of-the-art generalization performance on unseen domains. Further, we provide a thorough analysis to develop a holistic understanding of INDIGO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑