教会CLIP为序数回归培养数字感
Teach CLIP to Develop a Number Sense for Ordinal Regression
AI总结：
针对CLIP缺乏数字感而无法处理序数回归的问题，本文提出NumCLIP方法，通过粗分类与细预测两阶段设计及细粒度跨模态排序正则化损失，在三个序数回归任务上显著提升准确率。
摘要：
序数回归是计算机视觉领域的一个基础问题，通常需要针对特定任务定制并训练良好的模型。尽管预训练的视觉-语言模型（VLMs）在各种视觉任务上表现出色，但它们在序数回归方面的潜力尚未得到充分探索。在本研究中，我们首先考察了CLIP在序数回归上的潜力，期望该模型能够泛化到不同的序数回归任务和场景。遗憾的是，原始的CLIP在此任务上失败，因为当前的VLMs在封装诸如数字感之类的组合概念方面存在公认的局限性。我们提出了一种简单而有效的方法，称为NumCLIP，以提升VLMs的定量理解能力。我们将精确的图像到数字特定文本匹配问题拆解为粗分类和细预测两个阶段。我们用常见的语言概念对每个数值区间进行离散化和表述，以更好地利用CLIP中已有的预训练对齐。为了考虑序数回归固有的连续性质，我们提出了一种新颖的细粒度跨模态基于排序的正则化损失，专门设计用于在CLIP的特征空间中同时保持语义对齐和序数对齐。在三个通用序数回归任务上的实验结果证明了NumCLIP的有效性，在历史图像年代估计和图像美学评估任务上分别取得了10%和3.83%的准确率提升。代码已公开于https://github.com/xmed-lab/NumCLIP。
英文摘要：
Ordinal regression is a fundamental problem within the field of computer vision, with customised well-trained models on specific tasks. While pre-trained vision-language models (VLMs) have exhibited impressive performance on various vision tasks, their potential for ordinal regression has received less exploration. In this study, we first investigate CLIP's potential for ordinal regression, from which we expect the model could generalise to different ordinal regression tasks and scenarios. Unfortunately, vanilla CLIP fails on this task, since current VLMs have a well-documented limitation of encapsulating compositional concepts such as number sense. We propose a simple yet effective method called NumCLIP to improve the quantitative understanding of VLMs. We disassemble the exact image to number-specific text matching problem into coarse classification and fine prediction stages. We discretize and phrase each numerical bin with common language concept to better leverage the available pre-trained alignment in CLIP. To consider the inherent continuous property of ordinal regression, we propose a novel fine-grained cross-modal ranking-based regularisation loss specifically designed to keep both semantic and ordinal alignment in CLIP's feature space. Experimental results on three general ordinal regression tasks demonstrate the effectiveness of NumCLIP, with 10% and 3.83% accuracy improvement on historical image dating and image aesthetics assessment task, respectively. Code is publicly available at https://github.com/xmed-lab/NumCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
