arXiv 2508.041362026-02-03cs.CVcs.AI

UniFGVC: 一种通用无训练少样本细粒度视觉分类方法通过属性感知多模态检索

UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrieval

School of Traffic and Transportation, Beijing Jiaotong University（交通与运输学院，北京交通大学）；Foundation Modal Research Center, Institute of Automation, Chinese Academy of Sciences（基础模态研究中心，自动化研究所）；Queen Mary School Hainan, Beijing University of Posts and Telecommunications（海南女王学院，北京邮电大学）；Department of Computer Science, NUS School of Computing（计算机科学系，国立大学计算机学院）

Hongyu Guo, Xiangzhao Hao, Jiarui Guo, Haiyun Guo, Jinqiao Wang, Tat-Seng Chua

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

UniFGVC通过属性感知多模态检索方法，实现无训练少样本细粒度视觉分类，优于现有方法。

摘要：

少样本细粒度视觉分类（FGVC）旨在利用有限数据使模型能够区分微妙不同的类别。最近的研究大多通过微调预训练的视觉语言模型来获得性能提升，但会面临过拟合和泛化能力弱的问题。为了解决这个问题，我们引入了UniFGVC，一种通用无训练框架，将少样本FGVC重新表述为多模态检索。首先，我们提出了类别判别性视觉描述器（CDV-Captioner），利用多模态大语言模型（MLLMs）的开放世界知识生成结构化的文本描述，以捕捉区分密切相关类别的细粒度属性特征。CDV-Captioner使用思考链提示和视觉上相似的参考图像来减少幻觉并增强生成描述的判别性。利用它，我们可以将每张图像转换为图像-描述对，使特征表示更加全面，并使用少量样本构建多模态类别模板，以供后续的检索流程使用。然后，现成的视觉和文本编码器将查询和模板对嵌入到联合空间中，通过在联合空间中检索最接近的模板来完成FGVC。UniFGVC确保能够广泛兼容各种MLLMs和编码器，提供可靠的泛化能力和适应性，适用于各种少样本FGVC场景。在12个FGVC基准测试中的广泛实验显示，其在一致性上优于先前的少样本CLIP方法，甚至优于几种完全监督的MLLMs方法。

英文摘要：

Few-shot fine-grained visual classification (FGVC) aims to leverage limited data to enable models to discriminate subtly distinct categories. Recent works mostly finetuned the pre-trained visual language models to achieve performance gain, yet suffering from overfitting and weak generalization. To deal with this, we introduce UniFGVC, a universal training-free framework that reformulates few-shot FGVC as multimodal retrieval. First, we propose the Category-Discriminative Visual Captioner (CDV-Captioner) to exploit the open-world knowledge of multimodal large language models (MLLMs) to generate a structured text description that captures the fine-grained attribute features distinguishing closely related classes. CDV-Captioner uses chain-of-thought prompting and visually similar reference images to reduce hallucination and enhance discrimination of generated captions. Using it we can convert each image into an image-description pair, enabling more comprehensive feature representation, and construct the multimodal category templates using few-shot samples for the subsequent retrieval pipeline. Then, off-the-shelf vision and text encoders embed query and template pairs, and FGVC is accomplished by retrieving the nearest template in the joint space. UniFGVC ensures broad compatibility with diverse MLLMs and encoders, offering reliable generalization and adaptability across few-shot FGVC scenarios. Extensive experiments on 12 FGVC benchmarks demonstrate its consistent superiority over prior few-shot CLIP-based methods and even several fully-supervised MLLMs-based approaches.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。