Holistically understanding an object and its 3D movable parts through visual perception models is essential for enabling an autonomous agent to interact with the world. For autonomous driving, the dynamics and states of vehicle parts such as doors, the trunk, and the bonnet can provide meaningful semantic information and interaction states, which are essential to ensuring the safety of the self-driving vehicle. Existing visual perception models mainly focus on coarse parsing such as object bounding box detection or pose estimation and rarely tackle these situations. In this paper, we address this important autonomous driving problem by solving three critical issues. First, to deal with data scarcity, we propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images before reconstructing human-vehicle interaction (VHI) scenarios. Our approach is fully automatic without any human interaction, which can generate a large number of vehicles in uncommon states (VUS) for training deep neural networks (DNNs). Second, to perform fine-grained vehicle perception, we present a multi-task network for VUS parsing and a multi-stream network for VHI parsing. Third, to quantitatively evaluate the effectiveness of our data augmentation approach, we build the first VUS dataset in real traffic scenarios (e.g., getting on/out or placing/removing luggage). Experimental results show that our approach advances other baseline methods in 2D detection and instance segmentation by a big margin (over 8%). In addition, our network yields large improvements in discovering and understanding these uncommon cases. Moreover, we have released the source code, the dataset, and the trained model on Github (https://github.com/zongdai/EditingForDNN).
University of Toronto（多伦多大学）；Uber ATG（优步先进技术集团）；Brown University（布朗大学）
Self-driving vehicles must perceive and predict the future positions of nearby actors in order to avoid collisions and drive safely. A learned deep learning module is often responsible for this task, requiring large-scale, high-quality training datasets. As data collection is often significantly cheaper than labeling in this domain, the decision of which subset of examples to label can have a profound impact on model performance. Active learning techniques, which leverage the state of the current model to iteratively select examples for labeling, offer a promising solution to this problem. How…
OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（电子信息与电气工程学院，上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）；PhiGent Robotics, Beijing, China（PhiGent Robotics，北京，中国）
Collecting fine-grained labels usually requires expert-level domain knowledge and is prohibitive to scale up. In this paper, we propose Attribute Mix, a data augmentation strategy at attribute level to expand the fine-grained samples. The principle lies in that attribute features are shared among fine-grained sub-categories, and can be seamlessly transferred among images. Toward this goal, we propose an automatic attribute mining approach to discover attributes that belong to the same super-category, and Attribute Mix is operated by mixing semantically meaningful attribute features from two im…
University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research Asia（微软亚洲研究院）
Data augmentation is usually adopted to increase the amount of training data, prevent overfitting and improve the performance of deep models. However, in practice, random data augmentation, such as random image cropping, is low-efficiency and might introduce many uncontrolled background noises. In this paper, we propose Weakly Supervised Data Augmentation Network (WS-DAN) to explore the potential of data augmentation. Specifically, for each training image, we first generate attention maps to represent the object's discriminative parts by weakly supervised learning. Next, we augment the image g…
Department of Computer Science and Engineering University of Texas at Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Department of Computer Science and Engineering（计算机科学与工程系）；University of Texas at Arlington, Arlington, TX, USA（德克萨斯大学阿灵顿分校，阿灵顿，德克萨斯州，美国）；UTA（德克萨斯大学阿灵顿分校（UTA））
Fine-Grained Visual Categorization (FGVC) is a challenging topic in computer vision. It is a problem characterized by large intra-class differences and subtle inter-class differences. In this paper, we tackle this problem in a weakly supervised manner, where neural network models are getting fed with additional data using a data augmentation technique through a visual attention mechanism. We perform domain adaptive knowledge transfer via fine-tuning on our base network model. We perform our experiment on six challenging and commonly used FGVC datasets, and we show competitive improvement on ac…
Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
We propose a novel data augmentation method named 'FenceMask' that exhibits outstanding performance in various computer vision tasks. It is based on the 'simulation of object occlusion' strategy, which aim to achieve the balance between object occlusion and information retention of the input data. By enhancing the sparsity and regularity of the occlusion block, our augmentation method overcome the difficulty of small object augmentation and notably improve performance over baselines. Sufficient experiments prove the performance of our method is better than other simulate object occlusion appro…
Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangxi Zhuang Autonomous Region Information Center（广西壮族自治区信息中心）；The Hong Kong University of Science and Technology（香港科技大学）
Recently, the vision transformer (ViT) has made breakthroughs in image recognition. Its self-attention mechanism (MSA) can extract discriminative labeling information of different pixel blocks to improve image classification accuracy. However, the classification marks in their deep layers tend to ignore local features between layers. In addition, the embedding layer will be fixed-size pixel blocks. Input network Inevitably introduces additional image noise. To this end, we study a data augmentation vision transformer (DAVT) based on data augmentation and proposes a data augmentation method for…
Oxford Brookes University（牛津布鲁克斯大学）；University of Oxford（牛津大学）
In an autonomous driving system, perception - identification of features and objects from the environment - is crucial. In autonomous racing, high speeds and small margins demand rapid and accurate detection systems. During the race, the weather can change abruptly, causing significant degradation in perception, resulting in ineffective manoeuvres. In order to improve detection in adverse weather, deep-learning-based models typically require extensive datasets captured in such conditions - the collection of which is a tedious, laborious, and costly process. However, recent developments in Cycl…
From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院（ICT/CAS））；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（人工智能安全国家重点实验室，计算技术研究所，中国科学院（ICT/CAS））；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
Center for Data Science, Peking University（北京大学数据科学中心）；Alibaba Group（阿里巴巴集团）；CASIA；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理大学）；Salesforce Research Asia（赛富时亚洲研究院）
This paper investigates an open research problem of generating text-image pairs to improve the training of fine-grained image-to-text cross-modal retrieval task, and proposes a novel framework for paired data augmentation by uncovering the hidden semantic information of StyleGAN2 model. Specifically, we first train a StyleGAN2 model on the given dataset. We then project the real images back to the latent space of StyleGAN2 to obtain the latent codes. To make the generated images manipulatable, we further introduce a latent space alignment module to learn the alignment between StyleGAN2 latent …
State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；ByteDance, LarkAI（字节跳动 LarkAI）；Shanghai Key Lab of Intelligent Information Processing（上海智能信息处理重点实验室）
Fine-grained person perception such as body segmentation and pose estimation has been achieved with many 2D and 3D sensors such as RGB/depth cameras, radars (e.g., RF-Pose) and LiDARs. These sensors capture 2D pixels or 3D point clouds of person bodies with high spatial resolution, such that the existing Convolutional Neural Networks can be directly applied for perception. In this paper, we take one step forward to show that fine-grained person perception is possible even with 1D sensors: WiFi antennas. To our knowledge, this is the first work to perceive persons with pervasive WiFi devices, w…
Fudan University（复旦大学）；East China Normal University（华东师范大学）；Videt Lab（Videt实验室）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）
Document layout analysis (DLA) plays an important role in information extraction and document understanding. At present, document layout analysis has reached a milestone achievement, however, document layout analysis of non-Manhattan is still a challenge. In this paper, we propose an image layer modeling method to tackle this challenge. To measure the proposed image layer modeling method, we propose a manually-labeled non-Manhattan layout fine-grained segmentation dataset named FPD. As far as we know, FPD is the first manually-labeled non-Manhattan layout fine-grained segmentation dataset. To …
While the fine-grained visual categorization (FGVC) problems have been greatly developed in the past years, the Ultra-fine-grained visual categorization (Ultra-FGVC) problems have been understudied. FGVC aims at classifying objects from the same species (very similar categories), while the Ultra-FGVC targets at more challenging problems of classifying images at an ultra-fine granularity where even human experts may fail to identify the visual difference. The challenges for Ultra-FGVC mainly comes from two aspects: one is that the Ultra-FGVC often arises overfitting problems due to the lack of …
PP-SSL : Priority-Perception Self-Supervised Learning for Fine-Grained Recognition
College of Computer Science and Technology, Ocean University of China（中国海洋大学计算机科学与技术学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Computer Science, The University of Adelaide（阿德莱德大学计算机科学学院）；School of Information Science and Engineering, Shandong University（山东大学信息科学与工程学院）
Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation
School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；State Key Laboratory of Information Security, Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所信息安全国家重点实验室）；Beihang University（北京航空航天大学）；University of Chinese Academy of Sciences（中国科学院大学）；School of Cyber Science and Technology, Sun Yat-Sen University（中山大学网络空间科学与技术学院）
EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；The Hong Kong University of Science and Technology（香港科技大学）；LMMs-Lab；Beijing Institute of Technology（北京理工大学）；University of New South Wales（新南威尔士大学）
Despite the increasing popularity of LiDAR sensors, perception algorithms using 3D LiDAR data struggle with the 'sensor-bias problem'. Specifically, the performance of perception algorithms significantly drops when an unseen specification of LiDAR sensor is applied at test time due to the domain discrepancy. This paper presents a fast and flexible LiDAR augmentation method for the semantic segmentation task, called 'LiDomAug'. It aggregates raw LiDAR scans and creates a LiDAR scan of any configurations with the consideration of dynamic distortion and occlusion, resulting in instant domain augm…
Data augmentation is a commonly used approach to improving the generalization of deep learning models. Recent works show that learned data augmentation policies can achieve better generalization than hand-crafted ones. However, most of these works use unified augmentation policies for all samples in a dataset, which is observed not necessarily beneficial for all labels in multi-label classification tasks, i.e., some policies may have negative impacts on some labels while benefitting the others. To tackle this problem, we propose a novel Label-Based AutoAugmentation (LB-Aug) method for multi-la…
APPO: Attention-guided Perception Policy Optimization for Video Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing（中国人民大学北京校区人工智能学院）；AI Technology Center, Online Video Business Unit, Tencent PCG（腾讯PCG人工智能技术中心）
EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery
School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；Wenzhou Medical University（温州医科大学）；School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）
Fine-grained classification remains a challenging task because distinguishing categories needs learning complex and local differences. Diversity in the pose, scale, and position of objects in an image makes the problem even more difficult. Although the recent Vision Transformer models achieve high performance, they need an extensive volume of input data. To encounter this problem, we made the best use of GAN-based data augmentation to generate extra dataset instances. Oxford-IIIT Pets was our dataset of choice for this experiment. It consists of 37 breeds of cats and dogs with variations in sc…
