arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.08400cs.CV
收藏

面向开放世界视觉识别的大语言模型

Towards Open-Ended Visual Recognition with Large Language Model

ByteDance(字节跳动)

Qihang Yu, Xiaohui Shen, Liang-Chieh Chen

AI总结:

针对开放世界视觉识别中依赖类别名称和人工干预的问题,提出基于大语言模型的掩码分类器OSM,以生成方式预测标签,支持跨数据集训练,结合掩码提议模型在基准上取得良好效果。

摘要:

在开放式的物理世界中,定位并识别物体是机器感知领域长期存在的挑战。近期方法尝试通过使用类别无关的掩码(或框)提议模型,并辅以利用预提取文本嵌入的开放词汇分类器(如CLIP)来解决该问题。然而,值得注意的是,这些开放词汇识别模型在实际应用中仍存在局限性。一方面,它们在测试时依赖提供类别名称,识别性能很大程度上取决于用户预定义的语义类别集合。另一方面,当使用多个数据集进行训练时,需要人工干预以缓解它们之间的标签定义冲突。在本文中,我们引入了OmniScient模型(OSM),一种新颖的基于大语言模型(LLM)的掩码分类器,作为解决上述挑战的直接而有效的方案。具体而言,OSM以生成方式预测类别标签,从而在训练和测试时都无需提供类别名称。它还支持无需任何人工干预的跨数据集训练,并凭借从LLM获得的世界知识展现出强大的泛化能力。通过将OSM与现成的掩码提议模型相结合,我们在各种基准上展示了有前景的结果,并证明了其在处理新概念方面的有效性。代码/模型可在https://github.com/bytedance/OmniScient-Model获取。

英文摘要:

Localizing and recognizing objects in the open-ended physical world poses a long-standing challenge within the domain of machine perception. Recent methods have endeavored to address the issue by employing a class-agnostic mask (or box) proposal model, complemented by an open-vocabulary classifier (e.g., CLIP) using pre-extracted text embeddings. However, it is worth noting that these open-vocabulary recognition models still exhibit limitations in practical applications. On one hand, they rely on the provision of class names during testing, where the recognition performance heavily depends on this predefined set of semantic classes by users. On the other hand, when training with multiple datasets, human intervention is required to alleviate the label definition conflict between them. In this paper, we introduce the OmniScient Model (OSM), a novel Large Language Model (LLM) based mask classifier, as a straightforward and effective solution to the aforementioned challenges. Specifically, OSM predicts class labels in a generative manner, thus removing the supply of class names during both training and testing. It also enables cross-dataset training without any human interference, exhibiting robust generalization capabilities due to the world knowledge acquired from the LLM. By combining OSM with an off-the-shelf mask proposal model, we present promising results on various benchmarks, and demonstrate its effectiveness in handling novel concepts. Code/model are available at https://github.com/bytedance/OmniScient-Model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑