Multi-task convolutional neural networks (CNNs) have shown impressive results for certain combinations of tasks, such as single-image depth estimation (SIDE) and semantic segmentation. This is achieved by pushing the network towards learning a robust representation that generalizes well to different atomic tasks. We extend this concept by adding auxiliary tasks, which are of minor relevance for the application, to the set of learned tasks. As a kind of additional regularization, they are expected to boost the performance of the ultimately desired main tasks. To study the proposed approach, we picked vision-based road scene understanding (RSU) as an exemplary application. Since multi-task learning requires specialized datasets, particularly when using extensive sets of tasks, we provide a multi-modal dataset for multi-task RSU, called synMT. More than 2.5 $\cdot$ 10^5 synthetic images, annotated with 21 different labels, were acquired from the video game Grand Theft Auto V (GTA V). Our proposed deep multi-task CNN architecture was trained on various combination of tasks using synMT. The experiments confirmed that auxiliary tasks can indeed boost network performance, both in terms of final results and training time.
Unprejudiced Training Auxiliary Tasks Makes Primary Better: A Multi-Task Learning Perspective
School of Computer Science and Technology, Harbin Institute of Technology (HIT)（哈尔滨工业大学计算机科学与技术学院）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
Minimizing the Effect of Noise and Limited Dataset Size in Image Classification Using Depth Estimation as an Auxiliary Task with Deep Multitask Learning
University of Toronto（多伦多大学）；The Hospital for Sick Children（病童医院）；Vector Institute（Vector研究所）
Multitask Auxiliary Network for Perceptual Quality Assessment of Non-Uniformly Distorted Omnidirectional Images
School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（江西财经大学计算与人工智能学院）；Jiangxi Provincial Key Laboratory of Multimedia Intelligent Processing（江西省多媒体智能处理重点实验室）；Harvard Medical School, Harvard University（哈佛大学医学院）
Deep multi-modal aggregation network for MR image reconstruction with auxiliary modality
National Center for Artificial Intelligence (NCAI), SDAIA（沙特数据与人工智能局国家人工智能中心（NCAI））；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Institute of High Performance Computing, Agency for Science, Technology and Research（科技研究局高性能计算研究所）
