ClipTTT: CLIP引导的测试时间训练帮助LVLMs看得更清楚
ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better
AI总结：
本文提出ClipTTT方法，通过CLIP模型的图像-文本对齐能力，在测试时引导LVLMs适应退化条件，减少幻觉并提升描述准确性。
摘要：
大型视觉-语言模型（LVLMs）在测试时容易产生幻觉，尤其是当视觉输入被损坏时。我们发现这种损坏会引入额外的分布偏移，显著放大现实应用中的幻觉率。为了解决这一问题，我们提出了CLIP引导的测试时间训练（ClipTTT），一种在单个测试样本上实时适应LVLMs以应对退化条件的方法。具体而言，我们利用预训练CLIP模型的图像-文本对齐强度作为稳定的引导信号，以识别可靠的自监督目标，从而实现快速适应而不改变基础LVLMs。在标准幻觉基准上进行的大量实验表明，ClipTTT有效缓解了幻觉，并在视觉损坏下提高了描述忠实度。
英文摘要：
Large vision-language models (LVLMs) tend to hallucinate, especially when visual inputs are corrupted at test time. We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications. To address this, we propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample. Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs. Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
