Natural Language (NL) descriptions can be one of the most convenient or the only way to interact with systems built to understand and detect city scale traffic patterns and vehicle-related events. In this paper, we extend the widely adopted CityFlow Benchmark with NL descriptions for vehicle targets and introduce the CityFlow-NL Benchmark. The CityFlow-NL contains more than 5,000 unique and precise NL descriptions of vehicle targets, making it the first multi-target multi-camera tracking with NL descriptions dataset to our knowledge. Moreover, the dataset facilitates research at the intersection of multi-object tracking, retrieval by NL descriptions, and temporal localization of events. In this paper, we focus on two foundational tasks: the Vehicle Retrieval by NL task and the Vehicle Tracking by NL task, which take advantage of the proposed CityFlow-NL benchmark and provide a strong basis for future research on the multi-target multi-camera tracking by NL description task.
NVIDIA（英伟达）；San Jose State University（圣何塞州立大学）；University of Washington（华盛顿大学）
Urban traffic optimization using traffic cameras as sensors is driving the need to advance state-of-the-art multi-target multi-camera (MTMC) tracking. This work introduces CityFlow, a city-scale traffic camera dataset consisting of more than 3 hours of synchronized HD videos from 40 cameras across 10 intersections, with the longest distance between two simultaneous cameras being 2.5 km. To the best of our knowledge, CityFlow is the largest-scale dataset in terms of spatial coverage and the number of cameras/videos in an urban environment. The dataset contains more than 200K annotated bounding …
This paper addresses the problem of tracking moving objects of variable appearance in challenging scenes rich with features and texture. Reliable tracking is of pivotal importance in surveillance applications. It is made particularly difficult by the nature of objects encountered in such scenes: these too change in appearance and scale, and are often articulated (e.g. humans). We propose a method which uses fast motion detection and segmentation as a constraint for both building appearance models and their robust propagation (matching) in time. The appearance model is based on sets of local ap…
Omni Survey for Multimodality Analysis in Visual Object Tracking
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；School of Computer Science and Technology, China University of Mining and Technology（中国矿业大学计算机科学与技术学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（Surrey 大学视觉、语音与信号处理中心）
Shanghai Grandhonor Information Technology Co.Ltd（上海宏景信息科技有限公司）；Nanjing University of Aeronautics and Astronautic（南京航空航天大学）；New York University（纽约大学）
Multiple Object Tracking (MOT) has been a useful yet challenging task in many real-world applications such as video surveillance, intelligent retail, and smart city. The challenge is how to model long-term temporal dependencies in an efficient manner. Some recent works employ Recurrent Neural Networks (RNN) to obtain good performance, which, however, requires a large amount of training data. In this paper, we proposed a novel tracking method that integrates the auto-tuning Kalman method for prediction and the Gated Recurrent Unit (GRU) and achieves a near-optimum with a small amount of trainin…
Multi-target multi-camera tracking (MTMCT), i.e., tracking multiple targets across multiple cameras, is a crucial technique for smart city applications. In this paper, we propose an effective and reliable MTMCT framework for vehicles, which consists of a traffic-aware single camera tracking (TSCT) algorithm, a trajectory-based camera link model (CLM) for vehicle re-identification (ReID), and a hierarchical clustering algorithm to obtain the cross camera vehicle trajectories. First, the TSCT, which jointly considers vehicle appearance, geometric features, and some common traffic scenarios, is p…
Australian National University（澳大利亚国立大学）；Tsinghua University（清华大学）
Data associations in multi-target multi-camera tracking (MTMCT) usually estimate affinity directly from re-identification (re-ID) feature distances. However, we argue that it might not be the best choice given the difference in matching scopes between re-ID and MTMCT problems. Re-ID systems focus on global matching, which retrieves targets from all cameras and all times. In contrast, data association in tracking is a local matching problem, since its candidates only come from neighboring locations and time frames. In this paper, we design experiments to verify such misfit between global re-ID …
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（电子通信研究院）
In this paper, we propose a novel framework for multi-target multi-camera tracking (MTMCT) of vehicles based on metadata-aided re-identification (MA-ReID) and the trajectory-based camera link model (TCLM). Given a video sequence and the corresponding frame-by-frame vehicle detections, we first address the isolated tracklets issue from single camera tracking (SCT) by the proposed traffic-aware single-camera tracking (TSCT). Then, after automatically constructing the TCLM, we solve MTMCT by the MA-ReID. The TCLM is generated from camera topological configuration to obtain the spatial and tempora…
Australian National University（澳大利亚国立大学）；Tsinghua University（清华大学）
Multi-target multi-camera tracking (MTMCT) systems track targets across cameras. Due to the continuity of target trajectories, tracking systems usually restrict their data association within a local neighborhood. In single camera tracking, local neighborhood refers to consecutive frames; in multi-camera tracking, it refers to neighboring cameras that the target may appear successively. For similarity estimation, tracking systems often adopt appearance features learned from the re-identification (re-ID) perspective. Different from tracking, re-ID usually does not have access to the trajectory c…
Faculty Of Computer And Information, Cairo University（开罗大学计算机与信息学院）
In recent years, artificial intelligence (AI) based on deep learning (DL) has sparked tremendous global interest. DL is widely used today and has expanded into various interesting areas. It is becoming more popular in cross-subject research, such as studies of smart city systems, which combine computer science with engineering applications. Human action detection is one of these areas. Human action detection is an interesting challenge due to its stringent requirements in terms of computing speed and accuracy. High-accuracy real-time object tracking is also considered a significant challenge. …
State Key Laboratory of Computer Science Institute of Software Chinese Academy of Sciences（中国科学院软件研究所计算机科学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Machine Intelligence Technology Lab Alibaba Group（阿里巴巴集团机器智能技术实验室）
Multi-Target Multi-Camera Tracking has a wide range of applications and is the basis for many advanced inferences and predictions. This paper describes our solution to the Track 3 multi-camera vehicle tracking task in 2021 AI City Challenge (AICITY21). This paper proposes a multi-target multi-camera vehicle tracking framework guided by the crossroad zones. The framework includes: (1) Use mature detection and vehicle re-identification models to extract targets and appearance features. (2) Use modified JDETracker (without detection module) to track single-camera vehicles and generate single-came…
A Robust Deep Networks based Multi-Object MultiCamera Tracking System for City Scale Traffic
COMSATS University Islamabad, Lahore Campus（COMSATS大学伊斯兰堡分校拉合尔校区）；Pakistan Navy Engineering College, National University of Sciences and Technology (NUST)（巴基斯坦国立科技大学巴基斯坦海军工程学院）
提出一种基于深度学习的多目标多摄像头跟踪框架，利用Mask R-CNN、迁移学习与Deep SORT等技术解决城市规模交通中的车辆跟踪难题，在AI City Challenge数据集上取得IDF1为0.8289的优异性能。
NVIDIA Corporation（NVIDIA公司）；Santa Clara University（圣克拉拉大学）；University at Albany, SUNY（纽约州立大学阿尔巴尼分校）；Iowa State University（爱荷华州立大学）；Woven by Toyota, Japan（日本丰田公司）；United Arab Emirates University（阿拉伯联合酋长国大学）；National Yang-Ming Chiao-Tung University（国家阳明交通大学）；University of Macau（澳门大学）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Johns Hopkins University（约翰霍普金斯大学）
第九届 AI City Challenge 设置四个赛道，涵盖 3D 多摄像头跟踪、交通安全视频问答、仓库空间推理和鱼眼摄像头道路目标检测，吸引 245 支队伍参赛，并通过严格评测框架推动多项任务达到新基准。
ReWorld-Track: A Recursive Event World Model for Language-Guided Multi-Camera Tracking
Huazhong University of Science and Technology（华中科技大学）；Jiangxi University of Water Resources and Electric Power（江西水利电力大学）；Zhongnan University of Economics and Law（中南财经政法大学）
This paper presents the learned techniques during the Video Analysis Module of the Master in Computer Vision from the Universitat Autònoma de Barcelona, used to solve the third track of the AI-City Challenge. This challenge aims to track vehicles across multiple cameras placed in multiple intersections spread out over a city. The methodology followed focuses first in solving multi-tracking in a single camera and then extending it to multiple cameras. The qualitative results of the implemented techniques are presented using standard metrics for video analysis such as mAP for object detection an…
University of Washington（华盛顿大学）；University of Tokyo（东京大学）；Microsoft Research（微软研究院）
Tracking of multiple objects is an important application in AI City geared towards solving salient problems related to safety and congestion in an urban environment. Frequent occlusion in traffic surveillance has been a major problem in this research field. In this challenge, we propose a model-based vehicle localization method, which builds a kernel at each patch of the 3D deformable vehicle model and associates them with constraints in 3D space. The proposed method utilizes shape fitness evaluation besides color information to track vehicle objects robustly and efficiently. To build 3D car m…
We present TrackNet, a method for Multi-Target Multi-Camera (MTMC) vehicle tracking from traffic video sequences. Cross-camera vehicle tracking has proved to be a challenging task due to perspective, scale and speed variance, as well occlusions and noise conditions. Our method is based on a modular approach that first detects vehicles frame-by-frame using Faster R-CNN, then tracks detections through single camera using Kalman filter, and finally matches tracks by a triplet metric learning strategy. We conduct experiments on TrackNet within the AI City Challenge framework, and present competiti…
Smart City applications such as intelligent traffic routing or accident prevention rely on computer vision methods for exact vehicle localization and tracking. Due to the scarcity of accurately labeled data, detecting and tracking vehicles in 3D from multiple cameras proves challenging to explore. We present a massive synthetic dataset for multiple vehicle tracking and segmentation in multiple overlapping and non-overlapping camera views. Unlike existing datasets, which only provide tracking ground truth for 2D bounding boxes, our dataset additionally contains perfect labels for 3D bounding bo…
Active Scout: Multi-Target Tracking Using Neural Radiance Fields in Dense Urban Environments
DEVCOM Army Research Laboratory（DEVCOM陆军研究实验室）；Department of Electrical & Systems Engineering and General Robotics, Automation, Sensing and Perception (GRASP) Laboratory at the University of Pennsylvania（电气与系统工程系及通用机器人、自动化、传感和感知（GRASP）实验室，宾夕法尼亚大学）
NVIDIA Corporation（英伟达公司）；Santa Clara University（圣克拉拉大学）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）；Australian National University（澳大利亚国立大学）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Iowa State University（爱荷华州立大学）；Boston University（波士顿大学）；University of Washington（华盛顿大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
The 6th edition of the AI City Challenge specifically focuses on problems in two domains where there is tremendous unlocked potential at the intersection of computer vision and artificial intelligence: Intelligent Traffic Systems (ITS), and brick and mortar retail businesses. The four challenge tracks of the 2022 AI City Challenge received participation requests from 254 teams across 27 countries. Track 1 addressed city-scale multi-target multi-camera (MTMC) vehicle tracking. Track 2 addressed natural-language-based vehicle track retrieval. Track 3 was a brand new track for naturalistic drivin…
University of Maryland（马里兰大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Multi-camera vehicle tracking is one of the most complicated tasks in Computer Vision as it involves distinct tasks including Vehicle Detection, Tracking, and Re-identification. Despite the challenges, multi-camera vehicle tracking has immense potential in transportation applications including speed, volume, origin-destination (O-D), and routing data generation. Several recent works have addressed the multi-camera tracking problem. However, most of the effort has gone towards improving accuracy on high-quality benchmark datasets while disregarding lower camera resolutions, compression artifact…
Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation
Optoelectronics Center, Viettel Aerospace Institute, Viettel Group（Viettel集团光学电子中心、 Viettel航空航天研究所）；University of Engineering and Technology, Vietnam National University（越南国家大学工程大学）；School of Electrical and Electronic Engineering, Hanoi University of Science and Technology（河内科学技术大学电子与电气工程学院）
Enhancing Multi-Camera People Tracking with Anchor-Guided Clustering and Spatio-Temporal Consistency ID Re-Assignment
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）；National Center for High-Performance Computing（国家高性能计算中心）
提出一种结合锚定引导聚类与时空一致性的多摄像机多人员跟踪方法，在CVPR AI City Challenge 2023上以95.36% IDF1获第一名。
Multi-object tracking (MOT) has been dominated by the use of track by detection approaches due to the success of convolutional neural networks (CNNs) on detection in the last decade. As the datasets and bench-marking sites are published, research direction has shifted towards yielding best accuracy on generic scenarios including re-identification (reID) of objects while tracking. In this study, we narrow the scope of MOT for surveillance by providing a dedicated dataset of pedestrians and focus on in-depth analyses of well performing multi-object trackers to observe the weak and strong sides o…
Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes
College of Computer Science and Software Engineering, Shenzhen University, China（深圳大学计算机科学与软件工程学院，中国）；Department of Computer Science, City University of Hong Kong, China（香港城市大学计算机科学系，中国）
NVIDIA Corporation（英伟达公司）；Santa Clara University（圣克拉拉大学）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）；Australian National University（澳大利亚国立大学）；Iowa State University（爱荷华州立大学）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Boston University（波士顿大学）；University of Washington（华盛顿大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In comparison with person re-identification (ReID), which has been widely studied in the research community, vehicle ReID has received less attention. Vehicle ReID is challenging due to 1) high intra-class variability (caused by the dependency of shape and appearance on viewpoint), and 2) small inter-class variability (caused by the similarity in shape and appearance between vehicles produced by different manufacturers). To address these challenges, we propose a Pose-Aware Multi-Task Re-Identification (PAMTRI) framework. This approach includes two innovations compared with previous methods. Fi…
NVIDIA Corporation（英伟达公司）；Santa Clara University（圣克拉拉大学）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）；Australian National University（澳大利亚国立大学）；Iowa State University（爱荷华州立大学）；Indian Institute of Technology Kanpur（印度坎pur理工学院）；Woven by Toyota（丰田编织公司）；United Arab Emirates University（阿拉伯联合酋长国大学）；National Yang-Ming Chiao-Tung University（国立阳明交通大学）；Johns Hopkins University（约翰霍普金斯大学）；Emirates Center for Mobility Research（阿联酋交通研究中心）
第八届AI City Challenge设置5个CV相关赛道，吸引全球47个国家和地区的726支团队参赛，刷新了多项基准，部分成果超越现有SOTA，推动了CV与AI在多领域的融合研究。
University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）；University of Ferrara（费拉拉大学）；University of Parma（帕尔马大学）
Combining Natural Language with Vision represents a unique and interesting challenge in the domain of Artificial Intelligence. The AI City Challenge Track 5 for Natural Language-Based Vehicle Retrieval focuses on the problem of combining visual and textual information, applied to a smart-city use case. In this paper, we present All You Can Embed (AYCE), a modular solution to correlate single-vehicle tracking sequences with natural language. The main building blocks of the proposed architecture are (i) BERT to provide an embedding of the textual descriptions, (ii) a convolutional backbone along…
C$^2$T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination
The State Key Laboratory of Internet of Things for Smart City, University of Macau（物联网智能城市国家重点实验室，澳门大学）；Wuhan University（武汉大学）；Hong Kong Polytechnic University（香港理工大学）；Computer and Information Science, University of Macau（澳门大学计算机与信息科学）
NVIDIA Corporation（英伟达公司）；Santa Clara University（圣克拉拉大学）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）；Australian National University（澳大利亚国立大学）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Lafayette College（拉斐特学院）；Iowa State University（爱荷华州立大学）；Boston University（波士顿大学）；University of Washington（华盛顿大学）；QCraft（轻舟智航）
The AI City Challenge was created with two goals in mind: (1) pushing the boundaries of research and development in intelligent video analysis for smarter cities use cases, and (2) assessing tasks where the level of performance is enough to cause real-world adoption. Transportation is a segment ripe for such adoption. The fifth AI City Challenge attracted 305 participating teams across 38 countries, who leveraged city-scale real traffic data and high-quality synthetic data to compete in five challenge tracks. Track 1 addressed video-based automatic vehicle counting, where the evaluation being …
University of Technology Sydney（悉尼科技大学）；Zhejiang University（浙江大学）
Vehicle search is one basic task for the efficient traffic management in terms of the AI City. Most existing practices focus on the image-based vehicle matching, including vehicle re-identification and vehicle tracking. In this paper, we apply one new modality, i.e., the language description, to search the vehicle of interest and explore the potential of this task in the real-world scenario. The natural language-based vehicle search poses one new challenge of fine-grained understanding of both vision and language modalities. To connect language and vision, we propose to jointly train the state…
Santa Clara University（圣克拉拉大学）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）；Iowa State University（爱荷华州立大学）；Woven by Toyota（丰田编织公司）；United Arab Emirates University（阿拉伯联合酋长国大学）；National Yang Ming Chiao Tung University（国立阳明交通大学）；University of Macau（澳门大学）；North Carolina State University（北卡罗来纳州立大学）；Columbia University（哥伦比亚大学）；Milestone Systems（里程碑系统公司）；Xi’an Jiaotong University（西安交通大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Traffic anomaly detection has played a crucial role in Intelligent Transportation System (ITS). The main challenges of this task lie in the highly diversified anomaly scenes and variational lighting conditions. Although much work has managed to identify the anomaly in homogenous weather and scene, few resolved to cope with complex ones. In this paper, we proposed a dual-modality modularized methodology for the robust detection of abnormal vehicles. We introduced an integrated anomaly detection framework comprising the following modules: background modeling, vehicle tracking with detection, mas…
