From Handcrafted to Deep Features for Pedestrian Detection: A Survey
Tianjin University（天津大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Linköping University（林雪平大学）；Inception Institute of Artificial Intelligence（因塞普申人工智能研究院）
Jiale Cao, Yanwei Pang, Jin Xie, Fahad Shahbaz Khan, Ling Shao
Pedestrian detection is an important but challenging problem in computer vision, especially in human-centric tasks. Over the past decade, significant improvement has been witnessed with the help of handcrafted features and deep features. Here we present a comprehensive survey on recent advances in pedestrian detection. First, we provide a detailed review of single-spectral pedestrian detection that includes handcrafted features based methods and deep features based approaches. For handcrafted features based methods, we present an extensive review of approaches and find that handcrafted features with large freedom degrees in shape and space have better performance. In the case of deep features based approaches, we split them into pure CNN based methods and those employing both handcrafted and CNN based features. We give the statistical analysis and tendency of these methods, where feature enhanced, part-aware, and post-processing methods have attracted main attention. In addition to single-spectral pedestrian detection, we also review multi-spectral pedestrian detection, which provides more robust features for illumination variance. Furthermore, we introduce some related datasets and evaluation metrics, and compare some representative methods. We conclude this survey by emphasizing open problems that need to be addressed and highlighting various future directions. Researchers can track an up-to-date list at https://github.com/JialeCao001/PedSurvey.
University of Electronic Science and Technology of China（电子科技大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Alibaba-Zhejiang University Joint Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Pedestrian detection has achieved great improvements with the help of Convolutional Neural Networks (CNNs). CNN can learn high-level features from input images, but the insufficient spatial resolution of CNN feature channels (feature maps) may cause a loss of information, which is harmful especially to small instances. In this paper, we propose a new pedestrian detection framework, which extends the successful RPN+BF framework to combine handcrafted features and CNN features. RoI-pooling is used to extract features from both handcrafted channels (e.g. HOG+LUV, CheckerBoards or RotatedFilters) …
Path prediction is a fundamental task for estimating how pedestrians or vehicles are going to move in a scene. Because path prediction as a task of computer vision uses video as input, various information used for prediction, such as the environment surrounding the target and the internal state of the target, need to be estimated from the video in addition to predicting paths. Many prediction approaches that include understanding the environment and the internal state have been proposed. In this survey, we systematically summarize methods of path prediction that take video as input and and ext…
Institute of Anthropomatics & Robotics（人机交互与机器人研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Hensoldt Optronics GmbH（亨索尔特光电有限公司）；Autonomous Systems（自主系统（研究所））；Daimler TSS（戴姆勒TSS公司）
Multispectral images consisting of aligned visual-optical (VIS) and thermal infrared (IR) image pairs are well-suited for practical applications like autonomous driving or visual surveillance. Such data can be used to increase the performance of pedestrian detection especially for weakly illuminated, small-scaled, or partially occluded instances. The current state-of-the-art is based on variants of Faster R-CNN and thus passes through two stages: a proposal generator network with handcrafted anchor boxes for object localization and a classification network for verifying the object category. In…
Tianjin University（天津大学）；Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences（中国科学院西安光学精密机械研究所）
Pedestrian detection based on the combination of Convolutional Neural Network (i.e., CNN) and traditional handcrafted features (i.e., HOG+LUV) has achieved great success. Generally, HOG+LUV are used to generate the candidate proposals and then CNN classifies these proposals. Despite its success, there is still room for improvement. For example, CNN classifies these proposals by the full-connected layer features while proposal scores and the features in the inner-layers of CNN are ignored. In this paper, we propose a unifying framework called Multilayer Channel Features (MCF) to overcome the dr…
A Comprehensive Survey on Deep Gait Recognition: Algorithms, Datasets and Challenges
The University of Hong Kong（香港大学）；Southern University of Science and Technology（南方科技大学）；University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Hong Kong Polytechnic University（香港理工大学）
The accuracy of a classifier, when performing Pattern recognition, is mostly tied to the quality and representativeness of the input feature vector. Feature Selection is a process that allows for representing information properly and may increase the accuracy of a classifier. This process is responsible for finding the best possible features, thus allowing us to identify to which class a pattern belongs. Feature selection methods can be categorized as Filters, Wrappers, and Embed. This paper presents a survey on some Filters and Wrapper methods for handcrafted feature selection. Some discussio…
Department of Electrical Engineering and Computer Science, York University（约克大学电气工程与计算机科学系）；York University（约克大学）；Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；Toronto（多伦多）；ON, Canada（加拿大安大略省）
One of the major challenges that autonomous cars are facing today is driving in urban environments. To make it a reality, autonomous vehicles require the ability to communicate with other road users and understand their intentions. Such interactions are essential between the vehicles and pedestrians as the most vulnerable road users. Understanding pedestrian behavior, however, is not intuitive and depends on various factors such as demographics of the pedestrians, traffic dynamics, environmental conditions, etc. In this paper, we identify these factors by surveying pedestrian behavior studies,…
Walkable to Whom? Capturing Subjective Variability in Walkability Perception Using Multimodal Deep Learning
School of Civil and Environmental Engineering, University of New South Wales (UNSW)（新南威尔士大学土木与环境工程学院）；Research Centre for Integrated Transport Innovation (rCITI)（综合交通创新研究中心）
This paper presents an novel object type classification method for automotive applications which uses deep learning with radar reflections. The method provides object class information such as pedestrian, cyclist, car, or non-obstacle. The method is both powerful and efficient, by using a light-weight deep learning approach on reflection level radar data. It fills the gap between low-performant methods of handcrafted features and high-performant methods with convolutional neural networks. The proposed network exploits the specific characteristics of radar reflection data: It handles unordered …
Kyungpook National University（庆北国立大学）；Information Technology University（信息技术大学）；University of Warwick（华威大学）
In recent years visual object tracking has become a very active research area. An increasing number of tracking algorithms are being proposed each year. It is because tracking has wide applications in various real world problems such as human-computer interaction, autonomous vehicles, robotics, surveillance and security just to name a few. In the current study, we review latest trends and advances in the tracking area and evaluate the robustness of different trackers based on the feature extraction methods. The first part of this work comprises a comprehensive survey of the recently proposed t…
University of Oulu（奥卢大学）；Communication University of China（中国传媒大学）；SailYond Technology（赛扬德科技）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）
Face anti-spoofing (FAS) has lately attracted increasing attention due to its vital role in securing face recognition systems from presentation attacks (PAs). As more and more realistic PAs with novel types spring up, traditional FAS methods based on handcrafted features become unreliable due to their limited representation capacity. With the emergence of large-scale academic datasets in the recent decade, deep learning based FAS achieves remarkable performance and dominates this area. However, existing reviews in this field mainly focus on the handcrafted features, which are outdated and unin…
Feature representation and metric learning are two critical components in person re-identification models. In this paper, we focus on the feature representation and claim that hand-crafted histogram features can be complementary to Convolutional Neural Network (CNN) features. We propose a novel feature extraction model called Feature Fusion Net (FFN) for pedestrian image representation. In FFN, back propagation makes CNN features constrained by the handcrafted features. Utilizing color histogram features (RGB, HSV, YCbCr, Lab and YIQ) and texture features (multi-scale and multi-orientation Gab…
Occluded Person Re-Identification with Deep Learning: A Survey and Perspectives
Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）；School of Software, Xinjiang University（新疆大学软件学院）；School of Information Technology, Halmstad University（哈尔姆斯塔德大学信息技术学院）
A Survey on SAR ship classification using Deep Learning
PhD School in Computer Science, University of Pisa（计算机科学博士学院，比萨大学）；Institute of Information Science and Technologies, National Research Council of Italy（意大利国家研究委员会信息科学与技术研究所）；National Biodiversity Future Center - NBFC（国家生物多样性未来中心 - NBFC）
University of Maryland（马里兰大学）；Carnegie Mellon University（卡内基梅隆大学）；New York University（纽约大学）；IBM Research（IBM研究院）
Blind people have limited access to information about their surroundings, which is important for ensuring one's safety, managing social interactions, and identifying approaching pedestrians. With advances in computer vision, wearable cameras can provide equitable access to such information. However, the always-on nature of these assistive technologies poses privacy concerns for parties that may get recorded. We explore this tension from both perspectives, those of sighted passersby and blind users, taking into account camera visibility, in-person versus remote experience, and extracted visual …
In this survey we present a complete landscape of joint object detection and pose estimation methods that use monocular vision. Descriptions of traditional approaches that involve descriptors or models and various estimation methods have been provided. These descriptors or models include chordiograms, shape-aware deformable parts model, bag of boundaries, distance transform templates, natural 3D markers and facet features whereas the estimation methods include iterative clustering estimation, probabilistic networks and iterative genetic matching. Hybrid approaches that use handcrafted feature …
Black Sesame Technology Inc.（黑芝麻智能科技有限公司）；Futurewei Technology Inc.（星网锐捷技术有限公司）
Since DARPA Grand Challenges (rural) in 2004/05 and Urban Challenges in 2007, autonomous driving has been the most active field of AI applications. Almost at the same time, deep learning has made breakthrough by several pioneers, three of them (also called fathers of deep learning), Hinton, Bengio and LeCun, won ACM Turin Award in 2019. This is a survey of autonomous driving technologies with deep learning methods. We investigate the major fields of self-driving systems, such as perception, mapping and localization, prediction, planning and control, simulation, V2X and safety etc. Due to the l…
Hefei University of Technology（合肥工业大学）；University of Louisiana at Lafayette（路易斯安那大学拉斐特分校）
Due to object detection's close relationship with video analysis and image understanding, it has attracted much research attention in recent years. Traditional object detection methods are built on handcrafted features and shallow trainable architectures. Their performance easily stagnates by constructing complex ensembles which combine multiple low-level image features with high-level context from object detectors and scene classifiers. With the rapid development in deep learning, more powerful tools, which are able to learn semantic, high-level, deeper features, are introduced to address the…
University of Campinas（坎皮纳斯大学）；University Rovira i Virgili（罗维拉-威尔吉利大学）；National University of Ireland Galway（爱尔兰国立高威大学）
Intelligent video-surveillance is currently an active research field in computer vision and machine learning techniques. It provides useful tools for surveillance operators and forensic video investigators. Person re-identification (PReID) is one among these tools. It consists of recognizing whether an individual has already been observed over a camera in a network or not. This tool can also be employed in various possible applications such as off-line retrieval of all the video-sequences showing an individual of interest whose image is given a query, and online pedestrian tracking over multip…
University of Campinas (UNICAMP)（坎皮纳斯大学（UNICAMP））；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）；University Rovira i Virgili（罗维拉-威尔吉利大学）
Intelligent video-surveillance (IVS) is currently an active research field in computer vision and machine learning and provides useful tools for surveillance operators and forensic video investigators. Person re-identification (PReID) is one of the most critical problems in IVS, and it consists of recognizing whether or not an individual has already been observed over a camera in a network. Solutions to PReID have myriad applications including retrieval of video-sequences showing an individual of interest or even pedestrian tracking over multiple camera views. Different techniques have been pr…
Modality-Aware Feature Matching in Visual and Vision-Language Applications: A Comprehensive Survey
School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（江西财经大学计算机与人工智能学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of Computing and Communications, Lancaster University（兰卡斯特大学计算机与通讯学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（新加坡资讯研究院，科技研究局（A*STAR））；Department of Automation, Tsinghua University（清华大学自动化系）
Advanced automotive active-safety systems, in general, and autonomous vehicles, in particular, rely heavily on visual data to classify and localize objects such as pedestrians, traffic signs and lights, and other nearby cars, to assist the corresponding vehicles maneuver safely in their environments. However, the performance of object detection methods could degrade rather significantly under challenging weather scenarios including rainy conditions. Despite major advancements in the development of deraining approaches, the impact of rain on object detection has largely been understudied, espec…
Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions
International Hellenic University（国际希腊大学）；University of Thessaly（色萨利大学）；Democritus University of Thrace（德谟克利特大学）；University of Peloponnese（伯罗奔尼撒大学）；Harokopio University of Athens（哈罗科皮奥大学）
Deep Learning in Palmprint Recognition-A Comprehensive Survey
College of Computer Science, Sichuan University（四川大学计算机学院）；School of Computer and Information, Hefei University of Technology（合肥工业大学计算机与信息学院）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；School of Electrical and Electronic Engineering, College of Engineering, Yonsei University（延世大学工学院电气与电子工程学院）
Evolution of ReID: From Early Methods to LLM Integration
Information Retrieval and Knowledge Management Research Lab, York University, Toronto, Canada（信息检索与知识管理研究实验室，约克大学，多伦多，加拿大）；Department of Electrical Engineering and Computer Science, York University, Toronto, Canada（电气工程与计算机科学系，约克大学，多伦多，加拿大）
University of Wuppertal（伍珀塔尔大学）；Ruhr-University Bochum（波鸿鲁尔大学）；CARIAD SE；Volkswagen AG（大众汽车集团）；Center of Advanced Internet Studies（高级互联网研究中心）
Deep neural networks (DNN) have made impressive progress in the interpretation of image data, so that it is conceivable and to some degree realistic to use them in safety critical applications like automated driving. From an ethical standpoint, the AI algorithm should take into account the vulnerability of objects or subjects on the street that ranges from "not at all", e.g. the road itself, to "high vulnerability" of pedestrians. One way to take this into account is to define the cost of confusion of one semantic category with another and use cost-based decision rules for the interpretation o…
A Survey of Deep Learning for Group-level Emotion Recognition
Nanjing Institute of Technology（南京工程学院）；Southeast University（东南大学）；Jiangsu University（江苏大学）；Flinders University（弗林德斯大学）；Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）
State Key Laboratory of Information Engineering in Surveying Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
The emerging drone aerial survey has the advantages of low cost, high efficiency, and flexible use. However, UAVs are often equipped with cheap POS systems and non-measurement cameras, and their flight attitudes are easily affected. How to realize the large-scale mapping of UAV image-free control supported by POS faces many technical problems. The most basic and important core technology is how to accurately realize the absolute orientation of images through advanced aerial triangulation technology. In traditional aerial triangulation, image matching algorithms are constrained to varying degre…
Remote Sensing Image Dehazing: A Systematic Review of Progress, Challenges, and Prospects
School of Artificial Intelligence and Computer Science（人工智能与计算机科学学院）；Josef Kittler Research Institute on Artificial Intelligence（乔塞夫·基特勒人工智能研究所）；School of Electronic Engineering（电子工程学院）；College of Artificial Intelligence（人工智能学院）；Aerospace Information Research Institute（航天信息研究所）；Department of Computer Science（计算机科学系）
University of Birjand（比尔詹德大学）；Mazandaran University of Science and Technology（马赞达兰科技大学）；University of Qom（库姆大学）；University of Genova（热那亚大学）；Deakin University（迪肯大学）；Bu-Ali Sina University（布-阿里·西纳大学）；Maragheh Faculty of Medical Sciences（马拉盖医学院）；Obuda University（奥布达大学）；Singapore University of Social Sciences（新加坡社会科学大学）；National Heart Centre Singapore（新加坡国家心脏中心）；Ngee Ann Polytechnic（义安理工学院）；Asia University（亚洲大学）
Myocardial infarction (MI) results in heart muscle injury due to receiving insufficient blood flow. MI is the most common cause of mortality in middle-aged and elderly individuals around the world. To diagnose MI, clinicians need to interpret electrocardiography (ECG) signals, which requires expertise and is subject to observer bias. Artificial intelligence-based methods can be utilized to screen for or diagnose MI automatically using ECG signals. In this work, we conducted a comprehensive assessment of artificial intelligence-based approaches for MI detection based on ECG as well as other bio…
