arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.03583cs.CV
收藏

面向长尾图像分类的文本引导混合增强方法

Text-Guided Mixup Towards Long-Tailed Image Categorization

University of Washington(华盛顿大学);Alibaba Group(阿里巴巴集团)

Richard Franklin, Jiawei Yao, Deyang Zhong, Qi Qian, Juhua Hu

AI总结:

本文提出文本引导混合增强方法,利用CLIP等预训练视觉-语言模型的文本编码器捕捉类别语义关系,以缓解长尾图像分类中少数类别样本不足的问题,并在基准任务上验证了有效性。

摘要:

在许多实际应用中,训练数据的类别标签频率分布可能呈现长尾分布,这对需要大量平衡数据来训练深度神经网络的传统方法构成了挑战。收集和标注数据以平衡类别标签分布既昂贵又耗时。许多现有的解决方案,如集成学习、重平衡策略或对深度神经网络进行微调,都受到少数类别样本不足这一固有问题的限制。近年来,像CLIP这样的视觉-语言模型通过捕捉图像和文本对之间视觉与语言特征的相似性,已被视为零样本或少样本学习的有效解决方案。考虑到大规模预训练的视觉-语言模型可能包含对少数类别有价值的辅助文本信息,我们提出利用文本监督来应对长尾学习的挑战。具体而言,我们提出了一种新颖的文本引导混合增强技术,利用预训练文本编码器识别出的类别间语义关系来帮助缓解长尾问题。我们在基准长尾任务上的实证研究证明了所提方法的有效性,并提供了理论保证。代码可在 https://github.com/rsamf/text-guided-mixup 获取。

英文摘要:

In many real-world applications, the frequency distribution of class labels for training data can exhibit a long-tailed distribution, which challenges traditional approaches of training deep neural networks that require heavy amounts of balanced data. Gathering and labeling data to balance out the class label distribution can be both costly and time-consuming. Many existing solutions that enable ensemble learning, re-balancing strategies, or fine-tuning applied to deep neural networks are limited by the inert problem of few class samples across a subset of classes. Recently, vision-language models like CLIP have been observed as effective solutions to zero-shot or few-shot learning by grasping a similarity between vision and language features for image and text pairs. Considering that large pre-trained vision-language models may contain valuable side textual information for minor classes, we propose to leverage text supervision to tackle the challenge of long-tailed learning. Concretely, we propose a novel text-guided mixup technique that takes advantage of the semantic relations between classes recognized by the pre-trained text encoder to help alleviate the long-tailed problem. Our empirical study on benchmark long-tailed tasks demonstrates the effectiveness of our proposal with a theoretical guarantee. Our code is available at https://github.com/rsamf/text-guided-mixup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑