一次性对齐与提示一切以实现通用视觉感知
Aligning and Prompting Everything All at Once for Universal Visual Perception
AI总结：
本文提出通用视觉感知模型APE，通过将定位重构为开放词汇检测、将语义与全景分割统一为代理实例学习，一次性对齐和提示图像中的一切，在160多个数据集上以单一权重达到领先性能。
摘要：
视觉基础模型近年来被探索用于构建通用视觉系统。然而，主流范式通过将实例级任务建模为对象-词语对齐，带来了沉重的跨模态交互，这在提示目标检测和视觉定位方面效果不佳。另一类聚焦于像素级任务的工作通常面临物体与背景类别之间较大的标注差距，并受到前景目标分割与背景类别分割之间相互干扰的影响。与主流方法形成鲜明对比的是，我们提出了APE，这是一种通用视觉感知模型，能够一次性对齐和提示图像中的一切，以实例级句子-对象匹配范式执行多种任务，即检测、分割和定位。具体而言，APE通过将语言引导的定位重新表述为开放词汇检测，推进了检测与定位的融合，从而在保持跨模态融合有效性的同时，高效地将模型提示扩展到数千个类别词汇和区域描述。为了弥合不同像素级任务之间的粒度差距，APE将语义分割和全景分割统一为代理实例学习，将任何孤立区域视为独立实例。APE在具有自然且具有挑战性特征的大规模数据上一次性对齐视觉与语言表示，无需针对特定任务进行微调。在超过160个数据集上的大量实验表明，仅使用一套权重，APE就优于（或与）最先进模型（持平），证明了一种有效且通用的、面向任意对齐与提示的感知确实是可行的。代码和训练模型已发布在 https://github.com/shenyunhang/APE。
英文摘要：
Vision foundation models have been explored recently to build general-purpose vision systems. However, predominant paradigms, driven by casting instance-level tasks as an object-word alignment, bring heavy cross-modality interaction, which is not effective in prompting object detection and visual grounding. Another line of work that focuses on pixel-level tasks often encounters a large annotation gap of things and stuff, and suffers from mutual interference between foreground-object and background-class segmentation. In stark contrast to the prevailing methods, we present APE, a universal visual perception model for aligning and prompting everything all at once in an image to perform diverse tasks, i.e., detection, segmentation, and grounding, as an instance-level sentence-object matching paradigm. Specifically, APE advances the convergence of detection and grounding by reformulating language-guided grounding as open-vocabulary detection, which efficiently scales up model prompting to thousands of category vocabularies and region descriptions while maintaining the effectiveness of cross-modality fusion. To bridge the granularity gap of different pixel-level tasks, APE equalizes semantic and panoptic segmentation to proxy instance learning by considering any isolated regions as individual instances. APE aligns vision and language representation on broad data with natural and challenging characteristics all at once without task-specific fine-tuning. The extensive experiments on over 160 datasets demonstrate that, with only one-suit of weights, APE outperforms (or is on par with) the state-of-the-art models, proving that an effective yet universal perception for anything aligning and prompting is indeed feasible. Codes and trained models are released at https://github.com/shenyunhang/APE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
