通过跨越模态间隙检索实现多模态无监督域泛化
Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap
AI总结：
针对多模态无监督域泛化，本文提出基于联合视觉-语言空间检索的框架，通过 paired k-means、自适应文本增强及两个附加组件提升跨域零样本性能。
摘要：
域泛化（DG）是一个重要问题，旨在共享标签空间的假设下，利用一个或多个源域学习一个可泛化到未见测试域的模型。然而，大多数 DG 方法都假定能够获得目标标签空间中的大量源数据；这一要求对许多实际应用而言过于严格，因为在这些应用中，获取与目标任务相同的标签空间成本高得令人难以承受。针对这一情形，我们研究多模态版本的无监督域泛化（MUDG）问题，该问题在微调期间使用一个与任务无关的大型无标签源数据集。我们的框架并不显式假定源数据集与目标任务之间存在任何关系；相反，它仅依赖这样一个前提：源数据集能够在联合视觉-语言空间中被准确且高效地搜索。我们在 MUDG 设定下作出三项贡献。首先，我们从理论上证明，由于文本查询与用于粗量化的图像质心之间距离较大，跨模态近似最近邻搜索会遭遇低召回率问题。据此，我们提出 paired k-means，这是一种简单的聚类算法，通过将质心存储在查询空间而非图像空间中来提高最近邻召回率。其次，我们提出一种面向目标标签的自适应文本增强方案，旨在提高零样本准确率并使检索到的图像数据多样化。最后，我们提出两个简单但有效的组件，以进一步提高下游目标准确率。我们在各自基准上与最先进的仅名称迁移、无源 DG 和零样本（ZS）方法进行比较，并在 20 个多样化数据集上展示了一致的准确率提升。代码可在 https://github.com/Chris210634/mudg 获取。
英文摘要：
Domain generalization (DG) is an important problem that learns a model which generalizes to unseen test domains leveraging one or more source domains, under the assumption of shared label spaces. However, most DG methods assume access to abundant source data in the target label space, a requirement that proves overly stringent for numerous real-world applications, where acquiring the same label space as the target task is prohibitively expensive. For this setting, we tackle the multimodal version of the unsupervised domain generalization (MUDG) problem, which uses a large task-agnostic unlabeled source dataset during finetuning. Our framework does not explicitly assume any relationship between the source dataset and target task. Instead, it relies only on the premise that the source dataset can be accurately and efficiently searched in a joint vision-language space. We make three contributions in the MUDG setting. Firstly, we show theoretically that cross-modal approximate nearest neighbor search suffers from low recall due to the large distance between text queries and the image centroids used for coarse quantization. Accordingly, we propose paired k-means, a simple clustering algorithm that improves nearest neighbor recall by storing centroids in query space instead of image space. Secondly, we propose an adaptive text augmentation scheme for target labels designed to improve zero-shot accuracy and diversify retrieved image data. Lastly, we present two simple but effective components to further improve downstream target accuracy. We compare against state-of-the-art name-only transfer, source-free DG and zero-shot (ZS) methods on their respective benchmarks and show consistent improvement in accuracy on 20 diverse datasets. Code is available: https://github.com/Chris210634/mudg
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
