The gap between simulation and the real-world restrains many machine learning breakthroughs in computer vision and reinforcement learning from being applicable in the real world. In this work, we tackle this gap for the specific case of camera-based navigation, formulating it as following a visual cue in the foreground with arbitrary backgrounds. The visual cue in the foreground can often be simulated realistically, such as a line, gate or cone. The challenge then lies in coping with the unknown backgrounds and integrating both. As such, the goal is to train a visual agent on data captured in an empty simulated environment except for this foreground cue and test this model directly in a visually diverse real world. In order to bridge this big gap, we show it's crucial to combine following techniques namely: Randomized augmentation of the fore- and background, regularization with both deep supervision and triplet loss and finally abstraction of the dynamics by using waypoints rather than direct velocity commands. The various techniques are ablated in our experimental results both qualitatively and quantitatively finally demonstrating a successful transfer from simulation to the real world.
Weakly and Self-Supervised Class-Agnostic Motion Prediction for Autonomous Driving
College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）；SenseTime Research, Hong Kong, China（时光科技研究院，香港，中国）
The problem of detecting changes in a scene and segmenting the foreground from background is still challenging, despite previous work. Moreover, new RGBD capturing devices include depth cues, which could be incorporated to improve foreground segmentation. In this work, we present a new nonparametric approach where a unified model mixes the device multiple information cues. In order to unify all the device channel cues, a new probabilistic depth data model is also proposed where we show how handle the inaccurate data to improve foreground segmentation. A new RGBD video dataset is presented in o…
This paper presents a novel framework in which video/image segmentation and localization are cast into a single optimization problem that integrates information from low level appearance cues with that of high level localization cues in a very weakly supervised manner. The proposed framework leverages two representations at different levels, exploits the spatial relationship between bounding boxes and superpixels as linear constraints and simultaneously discriminates between foreground and background at bounding box and superpixel level. Different from previous approaches that mainly rely on d…
Instance segmentation methods require large datasets with expensive and thus limited instance-level mask labels. Partially supervised instance segmentation aims to improve mask prediction with limited mask labels by utilizing the more abundant weak box labels. In this work, we show that a class agnostic mask head, commonly used in partially supervised instance segmentation, has difficulties learning a general concept of foreground for the weakly annotated classes using box supervision only. To resolve this problem we introduce an object mask prior (OMP) that provides the mask head with the gen…
Institute of Intelligent Machines, Chinese Academy of Sciences（中国科学院合肥智能机械研究所）；University of Science and Technology of China（中国科学技术大学）；Hefei Normal University（合肥师范学院）；Anhui Engineering Laboratory for Intelligent Driving Technology and Application（安徽省智能驾驶技术与应用工程实验室）；University of Stirling（斯特灵大学）；Anhui University（安徽大学）
Traditional saliency detection via Markov chain only considers boundaries nodes. However, in addition to boundaries cues, background prior and foreground prior cues play a complementary role to enhance saliency detection. In this paper, we propose an absorbing Markov chain based saliency detection method considering both boundary information and foreground prior cues. The proposed approach combines both boundaries and foreground prior cues through bidirectional Markov chain. Specifically, the image is first segmented into superpixels and four boundaries nodes (duplicated as virtual nodes) are …
Seoul National University（首尔国立大学）；NAVER AI Lab（NAVER人工智能实验室）；University of Tübingen（图宾根大学）；Sogang University（西江大学）
Weakly supervised semantic segmentation (WSSS) methods are often built on pixel-level localization maps obtained from a classifier. However, training on class labels only, classifiers suffer from the spurious correlation between foreground and background cues (e.g. train and rail), fundamentally bounding the performance of WSSS. There have been previous endeavors to address this issue with additional supervision. We propose a novel source of information to distinguish foreground from the background: Out-of-Distribution (OoD) data, or images devoid of foreground object classes. In particular, w…
Automatic Salient object detection has received tremendous attention from research community and has been an increasingly important tool in many computer vision tasks. This paper proposes a novel bottom-up salient object detection framework which considers both foreground and background cues. First, A series of background and foreground seeds are selected from an image reliably, and then used for calculation of saliency map separately. Next, a combination of foreground and background saliency map is performed. Last, a refinement step based on geodesic distance is utilized to enhance salient re…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
This paper presents a novel framework in which image cosegmentation and colocalization are cast into a single optimization problem that integrates information from low level appearance cues with that of high level localization cues in a very weakly supervised manner. In contrast to multi-task learning paradigm that learns similar tasks using a shared representation, the proposed framework leverages two representations at different levels and simultaneously discriminates between foreground and background at the bounding box and superpixel level using discriminative clustering. We show empirical…
In this paper, we propose an efficient and discriminative model for salient object detection. Our method is carried out in a stepwise mechanism based on both divergence background and compact foreground cues. In order to effectively enhance the distinction between nodes along object boundaries and the similarity among object regions, a graph is constructed by introducing the concept of virtual node. To remove incorrect outputs, a scheme for selecting background seeds and a method for generating compactness foreground regions are introduced, respectively. Different from prior methods, we calcul…
Phantom-Insight: Adaptive Multi-cue Fusion for Video Camouflaged Object Detection with Multimodal LLM
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
Indian Institute of Technology, Kharagpur（印度理工学院克勒格布尔分校）；Indian Institute of Science, Bangalore（印度科学学院班加罗尔分院）
Salient object detection has become an important task in many image processing applications. The existing approaches exploit background prior and contrast prior to attain state of the art results. In this paper, instead of using background cues, we estimate the foreground regions in an image using objectness proposals and utilize it to obtain smooth and accurate saliency maps. We propose a novel saliency measure called `foreground connectivity' which determines how tightly a pixel or a region is connected to the estimated foreground. We use the values assigned by this measure as foreground wei…
DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation
Department of Computer Science and Engineering, Lehigh University（计算机科学与工程系，莱恩大学）；Department of Computer Science（计算机科学系）；Engineering, Lehigh University（工程系，莱恩大学）；Qualcomm AI Research（高通人工智能研究）
Most existing RGB-D salient object detection (SOD) methods focus on the foreground region when utilizing the depth images. However, the background also provides important information in traditional SOD methods for promising performance. To better explore salient information in both foreground and background regions, this paper proposes a Bilateral Attention Network (BiANet) for the RGB-D SOD task. Specifically, we introduce a Bilateral Attention Module (BAM) with a complementary attention mechanism: foreground-first (FF) attention and background-first (BF) attention. The FF attention focuses o…
Beihang University（北京航空航天大学）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；Universitat Politecnica de Catalunya（加泰罗尼亚理工大学）；Microsoft（微软）
This paper introduces an unsupervised framework to extract semantically rich features for video representation. Inspired by how the human visual system groups objects based on motion cues, we propose a deep convolutional neural network that disentangles motion, foreground and background information. The proposed architecture consists of a 3D convolutional feature encoder for blocks of 16 frames, which is trained for reconstruction tasks over the first and last frames of the sequence. A preliminary supervised experiment was conducted to verify the feasibility of proposed method by training the …
FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglement
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences（西安光学精密机械研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Xi’an Jiaotong University（西安交通大学）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）
Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation
Shanghai Jiao Tong University（上海交通大学）；Yinwang Intelligent Technology Co., Ltd.（银望智能科技有限公司）；Fudan University（复旦大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Georgia Institute of Technology（佐治亚理工学院）
Shenzhen University（深圳大学）；Wenzhou-Kean University（温州肯恩大学）；Shenzhen Institute of Artificial Intelligence of Robotics of Society（深圳市人工智能与机器人研究院）
While class activation map (CAM) generated by image classification network has been widely used for weakly supervised object localization (WSOL) and semantic segmentation (WSSS), such classifiers usually focus on discriminative object regions. In this paper, we propose Contrastive learning for Class-agnostic Activation Map (C$^2$AM) generation only using unlabeled image data, without the involvement of image-level supervision. The core idea comes from the observation that i) semantic information of foreground objects usually differs from their backgrounds; ii) foreground objects with similar a…
University of Electronic Science and Technology of China（电子科技大学）
This paper studies few-shot segmentation, which is a task of predicting foreground mask of unseen classes by a few of annotations only, aided by a set of rich annotations already existed. The existing methods mainly focus the task on "\textit{how to transfer segmentation cues from support images (labeled images) to query images (unlabeled images)}", and try to learn efficient and general transfer module that can be easily extended to unseen classes. However, it is proved to be a challenging task to learn the transfer module that is general to various classes. This paper solves few-shot segment…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Qingdao Academy of Intelligent Industries（青岛智能产业技术研究院）
In this paper, we propose a robust change detection method for intelligent visual surveillance. This method, named M4CD, includes three major steps. Firstly, a sample-based background model that integrates color and texture cues is built and updated over time. Secondly, multiple heterogeneous features (including brightness variation, chromaticity variation, and texture variation) are extracted by comparing the input frame with the background model, and a multi-source learning strategy is designed to online estimate the probability distributions for both foreground and background. The three fea…
Decouple before Align: Visual Disentanglement Enhances Prompt Tuning
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（合作中位网创新中心，上海交通大学）；School of Artificial Intelligence, Shanghai Jiao Tong University（人工智能学院，上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Beijing Institute of Technology（北京理工大学）；A*STAR Centre for Frontier AI Research（A*STAR前沿人工智能研究中心）
Object-level Correlation for Few-Shot Segmentation
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Samsung Electronics (China) R&D Centre（三星电子（中国）研发中心）；Shanghai AI Laboratory（上海人工智能实验室）；Institute for Infocomm Research (I2R), A*STAR Singapore（资讯通信研究院（I2R），新加坡A*STAR）；Center for Machine Vision and Signal Analysis (CMVS), University of Oulu（图像与信号分析中心（CMVS），奥卢大学）
