arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.09128cs.CV
收藏

通过提示对任意内容进行标记化

Tokenize Anything via Prompting

Key Laboratory of Intelligent Information Processing, ICT, CAS(中国科学院计算技术研究所智能信息处理重点实验室);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);University of Chinese Academy of Sciences(中国科学院大学)

Ting Pan, Lulu Tang, Xinlong Wang, Shiguang Shan

AI总结:

本文提出一个可提示的统一模型,通过视觉提示实现任意区域的分割、识别与描述,利用大规模掩码和CLIP语义先验,在区域描述任务上创下新纪录。

摘要:

我们提出了一种统一的、可提示的模型,能够同时分割、识别和描述任意内容。与SAM不同,我们旨在通过视觉提示构建一种通用的区域表示。为了实现这一目标,我们使用大规模分割掩码(例如SA-1B掩码)以及来自预训练的具有50亿参数的CLIP模型的语义先验来训练一个可泛化的模型。具体来说,我们通过在每个掩码令牌中添加一个语义令牌来构建一个可提示的图像解码器。语义令牌负责在预定义的概念空间中学习语义先验。通过对掩码令牌的分割和语义令牌的概念预测进行联合优化,我们的模型展现出强大的区域识别和定位能力。例如,一个额外训练的3800万参数的因果文本解码器在Visual Genome区域描述任务上以164.7的CIDEr分数创下了新纪录。我们相信该模型可以作为一种通用的区域级图像标记器,能够为广泛的视觉感知任务编码通用区域上下文。代码和模型可在https://github.com/baaivision/tokenize-anything获取。

英文摘要:

We present a unified, promptable model capable of simultaneously segmenting, recognizing, and captioning anything. Unlike SAM, we aim to build a versatile region representation in the wild via visual prompting. To achieve this, we train a generalizable model with massive segmentation masks, \eg, SA-1B masks, and semantic priors from a pre-trained CLIP model with 5 billion parameters. Specifically, we construct a promptable image decoder by adding a semantic token to each mask token. The semantic token is responsible for learning the semantic priors in a predefined concept space. Through joint optimization of segmentation on mask tokens and concept prediction on semantic tokens, our model exhibits strong regional recognition and localization capabilities. For example, an additional 38M-parameter causal text decoder trained from scratch sets a new record with a CIDEr score of 164.7 on the Visual Genome region captioning task. We believe this model can be a versatile region-level image tokenizer, capable of encoding general-purpose region context for a broad range of visual perception tasks. Code and models are available at {\footnotesize \url{https://github.com/baaivision/tokenize-anything}}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑