Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation
Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所);Peng Cheng Laboratory(鹏城实验室);Georgia Institute of Technology(佐治亚理工学院);Picsart AI Research (PAIR)(Picsart人工智能研究院)
Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yunchao Wei, Humphrey Shi
Pre-trained vision-language models, e.g. CLIP, have been increasingly used to address the challenging Open-Vocabulary Segmentation (OVS) task, benefiting from their well-aligned vision-text embedding space. Typical solutions involve either freezing CLIP during training to unilaterally maintain its zero-shot capability, or fine-tuning CLIP vision encoder to achieve perceptual sensitivity to local regions. However, few of them incorporate vision-text collaborative optimization. Based on this, we propose the Content-Dependent Transfer to adaptively enhance each text embedding by interacting with the input image, which presents a parameter-efficient way to optimize the text representation. Besides, we additionally introduce a Representation Compensation strategy, reviewing the original CLIP-V representation as compensation to maintain the zero-shot capability of CLIP. In this way, the vision and text representation of CLIP are optimized collaboratively, enhancing the alignment of the vision-text feature space. To the best of our knowledge, we are the first to establish the collaborative vision-text optimizing mechanism within the OVS field. Extensive experiments demonstrate our method achieves superior performance on popular OVS benchmarks. In open-vocabulary semantic segmentation, our method outperforms the previous state-of-the-art approaches by +0.5, +2.3, +3.4, +0.4 and +1.1 mIoU, respectively on A-847, A-150, PC-459, PC-59 and PAS-20. Furthermore, in a panoptic setting on ADE20K, we achieve the performance of 27.1 PQ, 73.5 SQ, and 32.9 RQ. Code will be available at https://github.com/jiaosiyu1999/MAFT-Plus.git .
Global-Local Collaborative Inference with LLM for Lidar-Based Open-Vocabulary Detection
Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Meituan(美团);University of Science and Technology Beijing(北京科技大学);Hangzhou Innovation Institute, Beihang University(北京航空航天大学杭州创新研究院)
ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology(深圳理工大学计算机科学与人工智能学院);School of Artificial Intelligence, Nanchang University(南昌大学人工智能学院)
Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detection
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Polytechnic Institute, Zhejiang University(浙江大学 polytechnic 院);ZTE(ZTE 公司)
School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院)
Single Shot Self-Reliant Scene Text Spotter by Decoupled yet Collaborative Detection and Recognition
Harbin Institute of Technology at Shenzhen(哈尔滨工业大学(深圳));Baidu Inc(百度公司)
典型的文本识别遵循两阶段识别范式,首先检测文本实例的边界,然后在检测到的区域内执行文本识别。尽管这样的斑点范例的显着进步,一个重要的限制是,文本识别的性能在很大程度上取决于文本检测的精度,导致潜在的错误传播从检测到识别。在这项工作中,我们提出了Single Shot Self-Reliant Scene Text Spotter v2(SRSTS v2),它通过将识别与检测分离,同时协同优化两个任务来规避这一限制。具体来说,我们的SRSTS v2样本的代表性特征点周围的每个潜在的文本实例,并进行文本检测和识别并行引导这些采样点。因此,文本识别不再依赖于检测,从而减轻了从检测到识别的错误传播。此外,采样模块是在检测和识别的监督下学习的,这允许两个任务之间的协同优化和相互增强。受益于这种采样驱动的并发spotting框架,我们的方法是能够正确地识别文本实例,即使精确的文本边界是具有挑战性的检测。在四个基准上进行的大量实验表明,我们的方法与最先进的检举人相比毫不逊色。