面向跨模态视觉推理的测试时分布学习适配器
Test-time Distribution Learning Adapter for Cross-modal Visual Reasoning
AI总结：
针对现有VLP模型下游适配存在表征偏差、计算复杂度高及域泛化能力弱的问题，提出直接在测试阶段运行的TT-DNA，通过高斯分布建模少样本支持集特征并与CLIP预测残差融合，在人-物交互推理任务上性能显著优于SOTA。
摘要：
视觉-语言预训练（Vision-Language Pre-Trained, VLP）模型（如CLIP）在学习通用视觉表征方面已展现出显著成效。现有多种方法旨在借助VLP模型习得的知识，在监督有限的情况下将VLP模型高效适配到下游任务，但这些方法要么会引入有偏表征，要么计算复杂度较高，阻碍了其在CLIP模型微调中的有效性。此外，当模型在特定领域的专属数据上训练后，其向未知领域泛化的能力会下降。本文提出测试时分布学习适配器（Test-Time Distribution LearNing Adapter, TT-DNA），该适配器直接在测试阶段运行。具体而言，我们通过估计高斯分布来对少样本支持图像的视觉特征建模，以捕获支持集中的知识；将查询图像与支持图像特征分布之间的余弦相似度作为视觉适配器的预测结果；随后，视觉适配器的预测结果通过残差连接与原始CLIP的预测结果融合，得到最终预测。我们在人-物交互视觉推理任务上开展了大量实验，结果表明，所提出的TT-DNA大幅优于现有最先进方法。
英文摘要：
Vision-Language Pre-Trained (VLP) models, such as CLIP, have demonstrated remarkable effectiveness in learning generic visual representations. Several approaches aim to efficiently adapt VLP models to downstream tasks with limited supervision, aiming to leverage the acquired knowledge from VLP models. However, these methods suffer from either introducing biased representations or requiring high computational complexity, which hinders their effectiveness in fine-tuning the CLIP model. Moreover, when a model is trained on data specific to a particular domain, its ability to generalize to uncharted domains diminishes. In this work, we propose Test-Time Distribution LearNing Adapter (TT-DNA) which directly works during the testing period. Specifically, we estimate Gaussian distributions to model visual features of the few-shot support images to capture the knowledge from the support set. The cosine similarity between query image and the feature distribution of support images is used as the prediction of visual adapter. Subsequently, the visual adapter's prediction merges with the original CLIP prediction via a residual connection, resulting in the final prediction. Our extensive experimental results on visual reasoning for human object interaction demonstrate that our proposed TT-DNA outperforms existing state-of-the-art methods by large margins.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
