arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.19696cs.LGcs.CV
收藏

视觉-语言模型是强大的噪声标签检测器

Vision-Language Models are Strong Noisy Label Detectors

School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)

Tong Wei, Hao-Tian Li, Chun-Shu Li, Jiang-Xin Shi, Yu-Feng Li, Min-Ling Zhang

AI总结:

本文提出去噪微调框架DeFT,利用视觉-语言模型预训练对齐能力,通过学习正负文本提示构建噪声标签检测器,并结合参数高效微调,在多个噪声数据集上验证了其在噪声标签检测和图像分类中的有效性。

摘要:

近年来,关于视觉-语言模型微调的研究在各种下游任务中展现了出色的性能。然而,在实际应用中获取准确标注数据的挑战在微调过程中构成了重大障碍。为应对这一挑战,本文提出了一种名为DeFT的去噪微调框架,用于适配视觉-语言模型。DeFT利用在数百万辅助图像-文本对上预训练的文本与视觉特征的鲁棒对齐来筛除噪声标签。该框架通过为每个类别学习正文本提示和负文本提示来构建噪声标签检测器。正提示旨在揭示该类别的独特特征,而负提示则作为可学习的阈值,用于区分干净样本和噪声样本。我们采用参数高效微调来适配预训练的视觉编码器,以促进其与学习到的文本提示的对齐。作为一个通用框架,DeFT可以通过利用精心选择的干净样本,无缝地将许多预训练模型微调到下游任务。在七个合成和真实世界噪声数据集上的实验结果验证了DeFT在噪声标签检测和图像分类中的有效性。

英文摘要:

Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DeFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑