Pedestrian Detection: Domain Generalization, CNNs, Transformers and Beyond
Group 42（G42集团）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；National Center for Artificial Intelligence（国家人工智能中心）
Pedestrian detection is the cornerstone of many vision based applications, starting from object tracking to video surveillance and more recently, autonomous driving. With the rapid development of deep learning in object detection, pedestrian detection has achieved very good performance in traditional single-dataset training and evaluation setting. However, in this study on generalizable pedestrian detectors, we show that, current pedestrian detectors poorly handle even small domain shifts in cross-dataset evaluation. We attribute the limited generalization to two main factors, the method and the current sources of data. Regarding the method, we illustrate that biasness present in the design choices (e.g anchor settings) of current pedestrian detectors are the main contributing factor to the limited generalization. Most modern pedestrian detectors are tailored towards target dataset, where they do achieve high performance in traditional single training and testing pipeline, but suffer a degrade in performance when evaluated through cross-dataset evaluation. Consequently, a general object detector performs better in cross-dataset evaluation compared with state of the art pedestrian detectors, due to its generic design. As for the data, we show that the autonomous driving benchmarks are monotonous in nature, that is, they are not diverse in scenarios and dense in pedestrians. Therefore, benchmarks curated by crawling the web (which contain diverse and dense scenarios), are an efficient source of pre-training for providing a more robust representation. Accordingly, we propose a progressive fine-tuning strategy which improves generalization. Code and models can accessed at https://github.com/hasanirtiza/Pedestron.
Pedestrian trajectory prediction is an essential and challenging task for a variety of real-life applications such as autonomous driving and robotic motion planning. Besides generating a single future path, predicting multiple plausible future paths is becoming popular in some recent work on trajectory prediction. However, existing methods typically emphasize spatial interactions between pedestrians and surrounding areas but ignore the smoothness and temporal consistency of predictions. Our model aims to forecast multiple paths based on a historical trajectory by modeling multi-scale graph-bas…
School of Mechanical Engineering, University of Science and Technology Beijing（北京科技大学机械工程学院）
As a core technology of the autonomous driving system, pedestrian trajectory prediction can significantly enhance the function of active vehicle safety and reduce road traffic injuries. In traffic scenes, when encountering with oncoming people, pedestrians may make sudden turns or stop immediately, which often leads to complicated trajectories. To predict such unpredictable trajectories, we can gain insights into the interaction between pedestrians. In this paper, we present a novel generative method named Spatial Interaction Transformer (SIT), which learns the spatio-temporal correlation of p…
VIT-Ped: Visionary Intention Transformer for Pedestrian Behavior Analysis
German University in Cairo（埃及德国大学）；Faculty of Media Engineering and Technology（媒体工程与技术学院）；Computer Science and Engineering Department（计算机科学与工程系）；C-DRiVeS Lab（C-DRiVeS实验室）；Cognitive Driving Research in Vehicular Systems（车载系统认知驾驶研究）
SenseTime Research（商汤科技研究院）；National University of Singapore（新加坡国立大学）
Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. Th…
Leibniz University Hannover（汉诺威莱布尼茨大学）；University of Twente（特温特大学）；Clausthal University of Technology（克劳斯塔尔工业大学）
Predicting trajectories of pedestrians based on goal information in highly interactive scenes is a crucial step toward Intelligent Transportation Systems and Autonomous Driving. The challenges of this task come from two key sources: (1) complex social interactions in high pedestrian density scenarios and (2) limited utilization of goal information to effectively associate with past motion information. To address these difficulties, we integrate social forces into a Transformer-based stochastic generative model backbone and propose a new goal-based trajectory predictor called ForceFormer. Diffe…
Pedestrian detection has achieved great improvements in recent years, while complex occlusion handling is still one of the most important problems. To take advantage of the body parts and context information for pedestrian detection, we propose the part and context network (PCN) in this work. PCN specially utilizes two branches which detect the pedestrians through body parts semantic and context information, respectively. In the Part Branch, the semantic information of body parts can communicate with each other via recurrent neural networks. In the Context Branch, we adopt a local competition …
University of Adelaide（阿德莱德大学）；NICTA（国家信息与通信技术研究院）
Compared to other applications in computer vision, convolutional neural networks have under-performed on pedestrian detection. A breakthrough was made very recently by using sophisticated deep CNN models, with a number of hand-crafted features, or explicit occlusion handling mechanism. In this work, we show that by re-using the convolutional feature maps (CFMs) of a deep convolutional neural network (DCNN) model as image features to train an ensemble of boosted decision models, we are able to achieve the best reported accuracy without using specially designed learning algorithms. We empiricall…
Hyper-STTN: Hypergraph Augmented Spatial-Temporal Transformer Network for Trajectory Prediction
School of Applied and Creative Computing, Purdue University（普渡大学应用与创意计算学院）；Department of Computer Science and Department of Intelligent Systems Engineering, Indiana University Bloomington（印第安纳大学布卢明顿分校计算机科学系和智能系统工程系）；Department of Applied Artificial Intelligence, Sungkyunkwan University（成均馆大学应用人工智能系）
Forecasting the trajectory of pedestrians in shared urban traffic environments is still considered one of the challenging problems facing the development of autonomous vehicles (AVs). In the literature, this problem is often tackled using recurrent neural networks (RNNs). Despite the powerful capabilities of RNNs in capturing the temporal dependency in the pedestrians' motion trajectories, they were argued to be challenged when dealing with longer sequential data. Thus, in this work, we are introducing a framework based on the transformer networks that were shown recently to be more efficient …
TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs
Perception, Robotics and Intelligent Machines Research Group (PRIME)（感知、机器人与智能机器研究组（PRIME））；Department of Computer Science, Université de Moncton（蒙克顿大学计算机科学系）
In video surveillance, pedestrian retrieval (also called person re-identification) is a critical task. This task aims to retrieve the pedestrian of interest from non-overlapping cameras. Recently, transformer-based models have achieved significant progress for this task. However, these models still suffer from ignoring fine-grained, part-informed information. This paper proposes a multi-direction and multi-scale Pyramid in Transformer (PiT) to solve this problem. In transformer-based architecture, each pedestrian image is split into many patches. Then, these patches are fed to transformer laye…
Pedestrian detection benefits greatly from deep convolutional neural networks (CNNs). However, it is inherently hard for CNNs to handle situations in the presence of occlusion and scale variation. In this paper, we propose W$^3$Net, which attempts to address above challenges by decomposing the pedestrian detection task into \textbf{\textit{W}}here, \textbf{\textit{W}}hat and \textbf{\textit{W}}hether problem directing against pedestrian localization, scale prediction and classification correspondingly. Specifically, for a pedestrian instance, we formulate its feature by three steps. i) We gene…
Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction
IMT Nord Europe, University of Lille, CNRS UMR 9189 - CRIStAL（IMT 北欧洲、里尔大学、法国国家科学研究中心 UMR 9189 - CRIStAL）；University of Lille, CNRS UMR 9189 - CRIStAL（里尔大学、法国国家科学研究中心 UMR 9189 - CRIStAL）；Explain
Beihang University（北京航空航天大学）；Bioinformatics Institute, A*STAR（新加坡科技研究局生物信息学研究所）；National University of Singapore（新加坡国立大学）
A major bottleneck of pedestrian detection lies on the sharp performance deterioration in the presence of small-size pedestrians that are relatively far from the camera. Motivated by the observation that pedestrians of disparate spatial scales exhibit distinct visual appearances, we propose in this paper an active pedestrian detector that explicitly operates over multiple-layer neuronal representations of the input still image. More specifically, convolutional neural nets such as ResNet and faster R-CNNs are exploited to provide a rich and discriminative hierarchy of feature representations as…
Johns Hopkins University（约翰斯·霍普金斯大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair & in-depth comparisons between Transformers and CNNs, focusing on robustness evalu…
GTransPDM: A Graph-embedded Transformer with Positional Decoupling for Pedestrian Crossing Intention Prediction
Department of Traffic Information and Control Engineering, Jilin University（交通信息与控制工程系，吉林大学）；BIT-Barcelona Innovative Transportation Research Group, Civil Engineering School, UPC Barcelona Tech（巴塞罗那创新交通研究组，土木工程学院，UPC巴塞罗那技术学院）；Computer Vision Center (CVC), Computer Science Department, Universitat Autònoma de Barcelona (UAB)（计算机视觉中心（CVC），计算机科学系，巴塞罗那自治大学（UAB））
针对行人位置畸变及多模态时序建模难题，本文提出 GTransPDM，通过位置解耦与图嵌入 Transformer 融合姿态、位置和车辆运动特征，在 PIE 和 JAAD 上取得最优性能。
MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Research Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University of Sydney（悉尼大学）
This paper expands the strength of deep convolutional neural networks (CNNs) to the pedestrian attribute recognition problem by devising a novel attribute aware pooling algorithm. Existing vanilla CNNs cannot be straightforwardly applied to handle multi-attribute data because of the larger label space as well as the attribute entanglement and correlations. We tackle these challenges that hampers the development of CNNs for multi-attribute classification by fully exploiting the correlation between different attributes. The multi-branch architecture is adopted for fucusing on attributes at diffe…
Convolutional neural networks (CNNs) have been widely used in various vision tasks, e.g. image classification, semantic segmentation, etc. Unfortunately, standard 2D CNNs are not well suited for spherical signals such as panorama images or spherical projections, as the sphere is an unstructured grid. In this paper, we present Spherical Transformer which can transform spherical signals into vectors that can be directly processed by standard CNNs such that many well-designed CNNs architectures can be reused across tasks and datasets by pretraining. To this end, the proposed method first uses loc…
Xi’an Jiaotong-Liverpool University（西交利物浦大学）；University of Liverpool（利物浦大学）
Although deep-learning based methods for monocular pedestrian detection have made great progress, they are still vulnerable to heavy occlusions. Using multi-view information fusion is a potential solution but has limited applications, due to the lack of annotated training samples in existing multi-view datasets, which increases the risk of overfitting. To address this problem, a data augmentation method is proposed to randomly generate 3D cylinder occlusions, on the ground plane, which are of the average size of pedestrians and projected to multiple views, to relieve the impact of overfitting …
UC Santa Cruz（加州大学圣克鲁兹分校）；Johns Hopkins University（约翰斯·霍普金斯大学）
The recent success of Vision Transformers is shaking the long dominance of Convolutional Neural Networks (CNNs) in image recognition for a decade. Specifically, in terms of robustness on out-of-distribution samples, recent research finds that Transformers are inherently more robust than CNNs, regardless of different training setups. Moreover, it is believed that such superiority of Transformers should largely be credited to their self-attention-like architectures per se. In this paper, we question that belief by closely examining the design of Transformers. Our findings lead to three highly ef…
Understanding pedestrian crossing behavior is an essential goal in intelligent vehicle development, leading to an improvement in their security and traffic flow. In this paper, we developed a method called IntFormer. It is based on transformer architecture and a novel convolutional video classification model called RubiksNet. Following the evaluation procedure in a recent benchmark, we show that our model reaches state-of-the-art results with good performance ($\approx 40$ seq. per second) and size ($8\times $smaller than the best performing model), making it suitable for real-time usage. We a…
The goal of this document is to provide a pedagogical introduction to the main concepts underpinning the training of deep neural networks using gradient descent; a process known as backpropagation. Although we focus on a very influential class of architectures called "convolutional neural networks" (CNNs) the approach is generic and useful to the machine learning community as a whole. Motivated by the observation that derivations of backpropagation are often obscured by clumsy index-heavy narratives that appear somewhat mathemagical, we aim to offer a conceptually clear, vectorized description…
ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relations
Department of Computer Science, Durham University（杜伦大学计算机科学系）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）
Structured pruning greatly eases the deployment of large neural networks in resource-constrained environments. However, current methods either involve strong domain expertise, require extra hyperparameter tuning, or are restricted only to a specific type of network, which prevents pervasive industrial applications. In this paper, we undertake a simple and effective approach that can be easily applied to both vision transformers and convolutional neural networks. Specifically, we consider pruning as an evolution process of sub-network structures that inherit weights through reconstruction techn…
Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs
Faculty of Computing, Sabaragamuwa University of Sri Lanka（斯里兰卡萨巴拉加穆瓦大学计算机学院）；Faculty of Engineering, University of Moratuwa（莫拉图瓦大学工程学院）；School of Computing Technologies, RMIT University（RMIT大学计算技术学院）；School of Engineering & Digital Technologies, University of Southern Queensland（南昆士兰大学工程与数字技术学院）；School of Engineering & Technologies, UNSW（新南威尔士大学工程与技术学院）；Faculty of Science and Technology, Charles Darwin University（查尔斯达尔文大学科学技术学院）
Tran-GCN: A Transformer-Enhanced Graph Convolutional Network for Person Re-Identification in Monitoring Videos
Faculty of Computing, Universiti Teknologi Malaysia（马来西亚理工大学计算机学院）；Malaysia-Japan International Institute of Technology, Universiti Teknologi Malaysia（马来西亚理工大学马来西亚-日本国际技术学院）
AstraZeneca（阿斯利康）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Science for Life Laboratory（生命科学实验室）
Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis. Recently, vision transformers (ViTs) have appeared as a competitive alternative to CNNs, yielding similar levels of performance while possessing several interesting properties that could prove beneficial for medical imaging tasks. In this work, we explore whether it is time to move to transformer-based models or if we should keep working with CNNs - can we trivially switch to transformers? If so, what are the advantages and drawbacks of switching to ViTs for medical ima…
Vision Transformers (VTs) are becoming a valuable alternative to Convolutional Neural Networks (CNNs) when it comes to problems involving high-dimensional and spatially organized inputs such as images. However, their Transfer Learning (TL) properties are not yet well studied, and it is not fully known whether these neural architectures can transfer across different domains as well as CNNs. In this paper we study whether VTs that are pre-trained on the popular ImageNet dataset learn representations that are transferable to the non-natural image domain. To do so we consider three well-studied ar…
State of the art magnetic resonance (MR) image super-resolution methods (ISR) using convolutional neural networks (CNNs) leverage limited contextual information due to the limited spatial coverage of CNNs. Vision transformers (ViT) learn better global context that is helpful in generating superior quality HR images. We combine local information of CNNs and global information from ViTs for image super resolution and output super resolved images that have superior quality than those produced by state of the art methods. We include extra constraints through multiple novel loss functions that pres…
Autonomous driving is a challenging topic that requires complex solutions in perception tasks such as recognition of road, lanes, traffic signs or lights, vehicles and pedestrians. Through years of research, computer vision has grown capable of tackling these tasks with monocular detectors that can provide remarkable detection rates with relatively low processing times. However, the recent appearance of Convolutional Neural Networks (CNNs) has revolutionized the computer vision field and has made possible approaches to perform full pixel-wise semantic segmentation in times close to real time (…
The Chinese University of Hong Kong（香港中文大学）；Samsung AI Cambridge（三星剑桥人工智能中心）；Queen Mary University of London（伦敦玛丽女王大学）
Self-attention based models such as vision transformers (ViTs) have emerged as a very competitive architecture alternative to convolutional neural networks (CNNs) in computer vision. Despite increasingly stronger variants with ever-higher recognition accuracies, due to the quadratic complexity of self-attention, existing ViTs are typically demanding in computation and model size. Although several successful design choices (e.g., the convolutions and hierarchical multi-stage structure) of prior CNNs have been reintroduced into recent ViTs, they are still not sufficient to meet the limited resou…
The University of Hong Kong（香港大学）；Advanced Biomedical Instrumentation Centre（先进生物医学仪器中心）
In recent years, self-supervised monocular depth estimation has drawn much attention since it frees of depth annotations and achieved remarkable results on standard benchmarks. However, most of existing methods only focus on either daytime or nighttime images, thus their performance degrades on the other domain because of the large domain shift between daytime and nighttime images. To address this problem, in this paper we propose a two-branch network named GlocalFuse-Depth for self-supervised depth estimation of all-day images. The daytime and nighttime image in input image pair are fed into …
Monash University（莫纳什大学）；Oxford University（牛津大学）；Alibaba Group（阿里巴巴集团）；Sun Yat-sen University（中山大学）；DarkMatter AI Research（暗物质人工智能研究院）；Qilu University of Technology（齐鲁工业大学）；RMIT University（皇家墨尔本理工大学）
Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. Here, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing, which adaptively slice a part of network parameters for inputs with diverse difficulty levels, while keeping parameters stored statically and contiguously in hardware to prevent the extra burden of sparse computation. Based on thi…
Universität Wuppertal（伍珀塔尔大学）；Delphi Deutschland（德尔福德国公司）；Massachusetts Institute of Technology（麻省理工学院）
Convolutional neural networks (CNNs) have demonstrated their superiority in numerous computer vision tasks, yet their computational cost results prohibitive for many real-time applications such as pedestrian detection which is usually performed on low-consumption hardware. In order to alleviate this drawback, most strategies focus on using a two-stage cascade approach. Essentially, in the first stage a fast method generates a significant but reduced amount of high quality proposals that later, in the second stage, are evaluated by the CNN. In this work, we propose a novel detection pipeline th…
