OVMR：基于多模态参考的开放词汇识别
OVMR: Open-Vocabulary Recognition with Multi-Modal References
AI总结：
OVMR通过融合文本描述与示例图像生成多模态分类器，并引入基于偏好的细化模块，实现即插即用的开放词汇识别，在多种场景下优于现有方法。
摘要：
开放词汇识别面临的挑战在于，模型对其所应用的新类别没有任何线索。现有工作提出了不同方法将类别线索嵌入模型，例如通过少样本微调、向视觉-语言模型提供类别名称或文本描述。微调耗时且会降低泛化能力；文本描述可能存在歧义，无法刻画视觉细节。本文从不同角度处理开放词汇识别，参考由文本描述和示例图像组成的多模态线索。我们提出的方法名为OVMR，采用两个创新组件来追求更鲁棒的类别线索嵌入。首先通过动态地用图像示例补充文本描述来生成多模态分类器；然后应用基于偏好的细化模块融合单模态和多模态分类器，旨在缓解低质量示例图像或文本描述带来的问题。所提出的OVMR是一个即插即用模块，并且与从互联网随机抓取的示例图像配合良好。大量实验证明了OVMR的优异性能，例如它在各种场景和设置下均优于现有方法。代码公开于https://github.com/Zehong-Ma/OVMR。
英文摘要：
The challenge of open-vocabulary recognition lies in the model has no clue of new categories it is applied to. Existing works have proposed different methods to embed category cues into the model, \eg, through few-shot fine-tuning, providing category names or textual descriptions to Vision-Language Models. Fine-tuning is time-consuming and degrades the generalization capability. Textual descriptions could be ambiguous and fail to depict visual details. This paper tackles open-vocabulary recognition from a different perspective by referring to multi-modal clues composed of textual descriptions and exemplar images. Our method, named OVMR, adopts two innovative components to pursue a more robust category cues embedding. A multi-modal classifier is first generated by dynamically complementing textual descriptions with image exemplars. A preference-based refinement module is hence applied to fuse uni-modal and multi-modal classifiers, with the aim to alleviate issues of low-quality exemplar images or textual descriptions. The proposed OVMR is a plug-and-play module, and works well with exemplar images randomly crawled from the Internet. Extensive experiments have demonstrated the promising performance of OVMR, \eg, it outperforms existing methods across various scenarios and setups. Codes are publicly available at \href{https://github.com/Zehong-Ma/OVMR}{https://github.com/Zehong-Ma/OVMR}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
