Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Xidian University（西安电子科技大学）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Intel Labs China（英特尔中国研究院）；Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）；School of Mechanical Engineering and Automation, Beihang University（北京航空航天大学机械工程及自动化学院）
Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, Jingjing Liu
End-to-end autonomous driving has great potential in the transportation industry. However, the lack of transparency and interpretability of the automatic decision-making process hinders its industrial adoption in practice. There have been some early attempts to use attention maps or cost volume for better model explainability which is difficult for ordinary passengers to understand. To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. ADAPT jointly trains both the driving caption task and the vehicular control prediction task, through a shared video representation. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time. The code, models and data are available at https://github.com/jxbbb/ADAPT.
Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving
Institute for AI Industry Research (AIR)（人工智能产业研究院（AIR））；Xingjian College, Tsinghua University（清华大学行健书院）；OpenDriveLab, Shanghai AI Lab（上海人工智能实验室OpenDriveLab）；Mercedes-Benz Group China Ltd.（梅赛德斯-奔驰集团中国有限公司）
Image Captioning with Context-Aware Auxiliary Guidance
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；University of Science and Technology of China（中国科学技术大学）
Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；School of Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人学院和机器人视觉感知与控制技术国家工程研究中心，湖南大学）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（计算机与人工智能学院，西南交通大学）；CRRC Zhuzhou Institute Company Ltd.（中车株洲院有限公司）；Nanyang Technological University（南洋理工大学）
