发表机构
Guangzhou University; The Second Affiliated Hospital of Guangzhou University of Chinese Medicine; Jinan University; Pengcheng Laboratory（广州大学; 广州中医药大学第二附属医院; 暨南大学; 鹏城实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对视觉-语言模型测试时分布偏移导致的性能下降问题，提出局部间隔恢复框架，通过样本级受保护间隔恢复与流级双阶段稳定机制，在多数据集上实现优于现有基线的性能。
AI 中文摘要
CLIP等视觉-语言模型（VLMs）具备出色的零样本能力，但在意外的测试时分布偏移下性能常急剧下降。测试时适应（TTA）是有前景的解决方案，但对无标签测试流持续适应VLMs存在根本挑战：传统以Top-1为中心的更新会通过破坏相关类别间的局部语义几何结构强化错误，而迭代适应会加剧渐进式偏差累积，最终导致模型模式崩溃。为克服这些耦合缺陷，我们提出轻量的单步TTA框架——局部间隔恢复（LMR）。在样本层面，我们的受保护间隔恢复（PMR）目标通过屏蔽合理的近Top候选免受外部硬负样本干扰，恢复局部语义几何结构；同时，为应对流层面的退化，我们引入双阶段稳定机制，包含自适应间隔（AM）控制器与偏差校正（BC），以动态打破渐进式偏差累积并防止模式崩溃。在CIFAR-C、ImageNet-C及ImageNet变体上的大量实验表明，LMR在极具挑战性的小批量测试场景下，始终优于当前最优的TTA基线，证明其具备极强的鲁棒性与效率。我们的代码可在该URL获取。
英文摘要
Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.
CommentsAccepted by ACM MM 2026
