By design, average precision (AP) for object detection aims to treat all classes independently: AP is computed independently per category and averaged. On one hand, this is desirable as it treats all classes equally. On the other hand, it ignores cross-category confidence calibration, a key property in real-world use cases. Unfortunately, under important conditions (i.e., large vocabulary, high instance counts) the default implementation of AP is neither category independent, nor does it directly reward properly calibrated detectors. In fact, we show that on LVIS the default implementation produces a gameable metric, where a simple, un-intuitive re-ranking policy can improve AP by a large margin. To address these limitations, we introduce two complementary metrics. First, we present a simple fix to the default AP implementation, ensuring that it is independent across categories as originally intended. We benchmark recent LVIS detection advances and find that many reported gains do not translate to improvements under our new evaluation, suggesting recent improvements may arise from difficult to interpret changes to cross-category rankings. Given the importance of reliably benchmarking cross-category rankings, we consider a pooled version of AP (AP-Pool) that rewards properly calibrated detectors by directly comparing cross-category rankings. Finally, we revisit classical approaches for calibration and find that explicitly calibrating detectors improves state-of-the-art on AP-Pool by 1.7 points
MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering
School of Computer and Information Science, Qinghai Institute of Technology（青海理工学院计算机与信息科学学院）；School of Information Science and Technology, Harbin Institute of Technology (Shen Zhen)（哈尔滨工业大学（深圳）信息科学与技术学院）；School of Computer Science and Technology, Qinghai University（青海大学计算机科学与技术学院）；Eryuan Digital Technology Co., Ltd.（洱源数字科技有限公司）
Coop. Medianet Innovation Center（媒体网络协同创新中心）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
When trained at a sufficient scale, self-supervised learning has exhibited a notable ability to solve a wide range of visual or language understanding tasks. In this paper, we investigate simple, yet effective approaches for adapting the pre-trained foundation models to the downstream task of interest, namely, open-vocabulary semantic segmentation. To this end, we make the following contributions: (i) we introduce Fusioner, with a lightweight, transformer-based fusion module, that pairs the frozen visual representation with language concept through a handful of image segmentation data. As a co…
SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；School of Software Engineering, South China University of Technology（华南理工大学软件学院）
University of Mississippi（密西西比大学）；National University of Singapore（新加坡国立大学）；Independent Researcher（独立研究员）；ASUS Intelligent Cloud Services (AICS)（华硕智能云服务（AICS））
