The diversity of building architecture styles of global cities situated on various landforms, the degraded optical imagery affected by clouds and shadows, and the significant inter-class imbalance of roof types pose challenges for designing a robust and accurate building roof instance segmentor. To address these issues, we propose an effective framework to fulfill semantic interpretation of individual buildings with high-resolution optical satellite imagery. Specifically, the leveraged domain adapted pretraining strategy and composite dual-backbone greatly facilitates the discriminative feature learning. Moreover, new data augmentation pipeline, stochastic weight averaging (SWA) training and instance segmentation based model ensemble in testing are utilized to acquire additional performance boost. Experiment results show that our approach ranks in the first place of the 2023 IEEE GRSS Data Fusion Contest (DFC) Track 1 test phase ($mAP_{50}$:50.6\%). Note-worthily, we have also explored the potential of multimodal data fusion with both optical satellite imagery and SAR data.
University of Chinese Academy of Sciences（中国科学院大学）；Horizon Robotics（地平线机器人）；Harbin Institute of Technology（哈尔滨工业大学）
The target of 2D human pose estimation is to locate the keypoints of body parts from input 2D images. State-of-the-art methods for pose estimation usually construct pixel-wise heatmaps from keypoints as labels for learning convolution neural networks, which are usually initialized randomly or using classification models on ImageNet as their backbones. We note that 2D pose estimation task is highly dependent on the contextual relationship between image patches, thus we introduce a self-supervised method for pretraining 2D pose estimation networks. Specifically, we propose Heatmap-Style Jigsaw P…
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" model…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Click-based interactive image segmentation aims at extracting objects with a limited user clicking. A hierarchical backbone is the de-facto architecture for current methods. Recently, the plain, non-hierarchical Vision Transformer (ViT) has emerged as a competitive backbone for dense prediction tasks. This design allows the original ViT to be a foundation model that can be finetuned for downstream tasks without redesigning a hierarchical backbone for pretraining. Although this design is simple and has been proven effective, it has not yet been explored for interactive image segmentation. To fi…
Proposal-Contrastive Pretraining for Object Detection from Fewer Data
Université Paris-Saclay（巴黎-萨克雷大学）；CEA（法国原子能委员会）；Université Jean Monnet Saint-Etienne（圣艾蒂安让·莫内大学）；CNRS（法国国家科学研究中心）；Laboratoire Hubert Curien UMR 5516（于贝尔·居里安实验室）；Institut Universitaire de France (IUF)（法国大学研究院）
MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation
German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing（德国癌症研究中心（DKFZ）海德堡，医学影像计算部）；Faculty of Mathematics and Computer Science, Heidelberg University（海德堡大学数学与计算机科学学院）；HIDSS4Health - Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany（HIDSS4Health - 欧洲信息与数据科学健康学校，卡尔斯鲁厄/海德堡，德国）；Medical Faculty Heidelberg, Heidelberg University（海德堡大学医学学院）；National Center for Tumor Diseases (NCT), Heidelberg, Germany（肿瘤国家研究中心（NCT），海德堡，德国）；Helmholtz Imaging, German Cancer Research Center, Germany（海德堡成像，德国癌症研究中心，德国）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital（模式分析与学习组，放射肿瘤学部，海德堡大学医院）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
We propose bootstrapped masked autoencoders (BootMAE), a new approach for vision BERT pretraining. BootMAE improves the original masked autoencoders (MAE) with two core designs: 1) momentum encoder that provides online feature as extra BERT prediction targets; 2) target-aware decoder that tries to reduce the pressure on the encoder to memorize target-specific information in BERT pretraining. The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance. Therefore, w…
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）
In this technical report, we present our solution, dubbed MV-FCOS3D++, for the Camera-Only 3D Detection track in Waymo Open Dataset Challenge 2022. For multi-view camera-only 3D detection, methods based on bird-eye-view or 3D geometric representations can leverage the stereo cues from overlapped regions between adjacent views and directly perform 3D detection without hand-crafted post-processing. However, it lacks direct semantic supervision for 2D backbones, which can be complemented by pretraining simple monocular-based detectors. Our solution is a multi-view framework for 4D detection follo…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Cloud + AI（微软云与人工智能）
We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token. To address this issue, we develop the Cross-Shaped Window self-attention mechanism for computing self-attention in the horizontal and vertical stripes in parallel that form a cross-shaped window, with each stripe obtained by splitting the input feature into stripes of equal width. We provide a mat…
Pretraining is a dominant paradigm in computer vision. Generally, supervised ImageNet pretraining is commonly used to initialize the backbones of person re-identification (Re-ID) models. However, recent works show a surprising result that CNN-based pretraining on ImageNet has limited impacts on Re-ID system due to the large domain gap between ImageNet and person Re-ID data. To seek an alternative to traditional pretraining, here we investigate semantic-based pretraining as another method to utilize additional textual data against ImageNet pretraining. Specifically, we manually construct a dive…
DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining
Fordham University（福特汉姆大学）；Rutgers University（罗格斯大学）；Pioneer Centre for AI（先锋人工智能中心）；Cold Spring Harbor Laboratory（冷泉港实验室）；Brown University（布朗大学）；University of Copenhagen（哥本哈根大学）
TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation
SAIVT Group, School of Electrical Engineering and Robotics, Queensland University of Technology（澳大利亚昆士兰科技大学电气工程与机器人学院SAIVT组）；CSIRO Robotics, CSIRO（澳大利亚联邦科学与工业研究组织机器人部）
Exploring the Untouched Sweeps for Conflict-Aware 3D Segmentation Pretraining
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Department of Computer Science and Technology, Xiamen University（厦门大学计算机科学与技术系）
STRADAViT: Self-Supervised Domain Adaptation of Vision Transformer Backbones for Radio Astronomy
organization= Institute of Space Sciences；Astronomy , addressline= University of Malta , city= Msida , postcode= MSD2080 , country= Malta；organization= School of Physics；Astronomy, Royal Observatory , addressline= University of Edinburgh , city= Edinburgh , country= United Kingdom
GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives
XPeng Inc.（小鹏汽车）；Peking University（北京大学）；The University of Hong Kong（香港大学）；University of California, Berkeley（加州大学伯克利分校）；Princeton University（普林斯顿大学）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；HKUST (GZ)（香港科技大学（广州））
In recent years generative models of visual data have made a great progress, and now they are able to produce images of high quality and diversity. In this work we study representations learnt by a GAN generator. First, we show that these representations can be easily projected onto semantic segmentation map using a lightweight decoder. We find that such semantic projection can be learnt from just a few annotated images. Based on this finding, we propose LayerMatch scheme for approximating the representation of a GAN generator that can be used for unsupervised domain-specific pretraining. We c…
AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware
Department of Laboratory Medicine & Pathology, University of Washington（华盛顿大学检验医学与病理学系）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Electrical Engineering & Computer Science, Oregon State University（俄勒冈州立大学电气工程与计算机科学系）
