arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.00481cs.CV
收藏

视觉语言模型的高效开放集单图像测试时自适应

Efficient Open Set Single Image Test Time Adaptation of Vision Language Models

Indian Institute of Science(印度科学院)

Manogna Sreenivas, Soma Biswas

AI总结:

针对现实开放集场景下单图像测试时自适应问题,提出利用动态特征库和对比学习的ROSITA框架,有效区分已知与未知类别,在多个基准上实现性能与计算效率的双重提升。

摘要:

使模型适应以数据分布偏移和未见测试场景为特征的动态现实环境,是深度学习中的一个关键挑战。本文考虑了一种现实且具挑战性的测试时自适应(TTA)设置,模型必须持续适应逐个到达的测试样本,同时区分已知和未知类别。当前的TTA方法在封闭集假设或批处理下运行,不同于现实世界的开放集场景。我们通过建立使用视觉语言模型进行开放集单图像测试时自适应的综合基准来解决这一限制。此外,我们提出了ROSITA,这是一个利用动态更新的特征库识别可靠测试样本,并采用对比学习目标改善已知与未知类别分离的新框架。该方法有效地使模型适应已知类别的领域偏移,同时拒绝不熟悉的样本。在多个现实基准上的广泛实验表明,ROSITA在开放集TTA中确立了新的最先进水平,为实时部署实现了强大的性能和计算效率。

英文摘要:

Adapting models to dynamic, real-world environments characterized by shifting data distributions and unseen test scenarios is a critical challenge in deep learning. In this paper, we consider a realistic and challenging Test-Time Adaptation setting, where a model must continuously adapt to test samples that arrive sequentially, one at a time, while distinguishing between known and unknown classes. Current Test-Time Adaptation methods operate under closed-set assumptions or batch processing, differing from the real-world open-set scenarios. We address this limitation by establishing a comprehensive benchmark for {\em Open-set Single-image Test-Time Adaptation using Vision-Language Models}. Furthermore, we propose ROSITA, a novel framework that leverages dynamically updated feature banks to identify reliable test samples and employs a contrastive learning objective to improve the separation between known and unknown classes. Our approach effectively adapts models to domain shifts for known classes while rejecting unfamiliar samples. Extensive experiments across diverse real-world benchmarks demonstrate that ROSITA sets a new state-of-the-art in open-set TTA, achieving both strong performance and computational efficiency for real-time deployment. Our code can be found at the project site https://manogna-s.github.io/rosita/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑