Performing multiple heterogeneous visual tasks in dynamic scenes is a hallmark of human perception capability. Despite remarkable progress in image and video recognition via representation learning, current research still focuses on designing specialized networks for singular, homogeneous, or simple combination of tasks. We instead explore the construction of a unified model for major image and video recognition tasks in autonomous driving with diverse input and output structures. To enable such an investigation, we design a new challenge, Video Task Decathlon (VTD), which includes ten representative image and video tasks spanning classification, segmentation, localization, and association of objects and pixels. On VTD, we develop our unified network, VTDNet, that uses a single structure and a single set of weights for all ten tasks. VTDNet groups similar tasks and employs task interaction stages to exchange information within and between task groups. Given the impracticality of labeling all tasks on all frames, and the performance degradation associated with joint training of many tasks, we design a Curriculum training, Pseudo-labeling, and Fine-tuning (CPF) scheme to successfully train VTDNet on all tasks and mitigate performance loss. Armed with CPF, VTDNet significantly outperforms its single-task counterparts on most tasks with only 20% overall computations. VTD is a promising new direction for exploring the unification of perception tasks in autonomous driving.
123D: Unifying Multi-Modal Autonomous Driving Data at Scale
KE:SAI；University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；NVIDIA Research（NVIDIA研究）；Valeo Brain（法雷奥大脑）；OpenDriveLab at Shanghai Innovation Institute（上海创新研究院自动驾驶实验室）；Zhejiang University（浙江大学）；Delft University of Technology（代尔夫特理工大学）
DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers
Centre for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港中文大学信息工程中心人工智能与机器人中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
GraphWorld: Long-Horizon Planning with World Models for End-to-End Autonomous Driving
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院，交通数据挖掘与具身智能北京市重点实验室）；School of Artificial Intelligence (School of Software), Yanshan University（燕山大学人工智能学院（软件学院））；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）；University of Macau（澳门大学）；The University of Queensland（昆士兰大学）
Generative AI for Autonomous Driving: Frontiers and Opportunities
Texas A&M University（德克萨斯农工大学）；University of California, Riverside（加州大学河滨分校）；University of Michigan（密歇根大学）；Purdue University（普渡大学）；Columbia University（哥伦比亚大学）；University of Florida（佛罗里达大学）；Technische Universität München（慕尼黑技术大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Notre Dame（诺特丹大学）；Adobe Research（Adobe研究）；Lehigh University（莱斯利大学）；Cleveland State University（克利夫兰州立大学）；Johns Hopkins University（约翰霍普金斯大学）；University of California, Merced（加州大学默塞德分校）；University of Utah（犹他大学）；Texas A&M Transportation Institute（德克萨斯农工大学交通研究所）；Bosch Research North America（博世北美研究）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；NVIDIA（英伟达）；Arizona State University（亚利桑那州立大学）
Unifying Image Counterfactuals and Feature Attributions with Latent-Space Adversarial Attacks
Department of Statistics, University of California, Berkeley（加州大学伯克利分校统计学系）；Department of Statistics and Data Science, University of Pennsylvania（宾夕法尼亚大学统计学与数据科学系）
NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networks
Worcester Polytechnic Institute（沃斯特理工大学）；Texas A&M University（德克萨斯A&M大学）；Tohoku University（东北大学）；University of Michigan（密歇根大学）；Bosch Research North America & Bosch Center for AI（博世北美研究与人工智能中心）
