Optimizing the quality of result (QoR) and the quality of service (QoS) of AI-empowered autonomous systems simultaneously is very challenging. First, there are multiple input sources, e.g., multi-modal data from different sensors, requiring diverse data preprocessing, sensor fusion, and feature aggregation. Second, there are multiple tasks that require various AI models to run simultaneously, e.g., perception, localization, and control. Third, the computing and control system is heterogeneous, composed of hardware components with varied features, such as embedded CPUs, GPUs, FPGAs, and dedicated accelerators. Therefore, autonomous systems essentially require multi-modal multi-task (MMMT) learning which must be aware of hardware performance and implementation strategies. While MMMT learning has been attracting intensive research interests, its applications in autonomous systems are still underexplored. In this paper, we first discuss the opportunities of applying MMMT techniques in autonomous systems and then discuss the unique challenges that must be solved. In addition, we discuss the necessity and opportunities of MMMT model and hardware co-design, which is critical for autonomous systems especially with power/resource-limited or heterogeneous platforms. We formulate the MMMT model and heterogeneous hardware implementation co-design as a differentiable optimization problem, with the objective of improving the solution quality and reducing the overall power consumption and critical path latency. We advocate for further explorations of MMMT in autonomous systems and software/hardware co-design solutions.
M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving
School of Vehicle and Mobility, Tsinghua University(清华大学车辆与出行学院);SenseTime Research(商汤科技研究院);School of Mechanical Engineering, Beijing Institute of Technology(北京理工大学机械工程学院);School of Computer and Information Technology, Beijing Jiaotong University(北京交通大学计算机与信息技术学院)
Evaluation Is All You Need for Multi-Modal Autonomous Driving
School of Vehicle and Mobility & College of AI, Tsinghua University(清华大学车辆与运载学院与人工智能学院);Dongfeng Motor Corporation Research & Development Institute(东风汽车集团有限公司研发总院)
Hardware-Enforced Semantic Coordination for Safety-Critical Real-Time Autonomous Systems
Department of Computer Science University of the Bundeswehr Munich Neubiberg, Germany;Department of Computer Science Sapienza University of Rome Rome, Italy;STAKE Lab University of Molise Campobasso, Italy
Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs
University of Technology Sydney(悉尼科技大学);Australia Artificial Intelligence Institute(澳大利亚人工智能研究院);School of Computer Science(计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University(复旦大学脑科学智能科技研究院);School of Computer Science, Fudan University(复旦大学计算机科学学院)
Parameterized Decision-making with Multi-modal Perception for Autonomous Driving
University of Electronic Science and Technology of China(电子科技大学);Yangtze Delta Region Institute(Quzhou), University of Electronic Science and Technology of China(电子科技大学长三角研究院(衢州));University of Michigan(密歇根大学)
Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences(中国科学院大学杭州高等研究院);Shanghai Jiao Tong University(上海交通大学)
H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);School of Artificial Intelligence, University of Chinese Academy of Science(中国科学院大学人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong Polytechnic University(香港理工大学)
CORENet: Cross-Modal 4D Radar Denoising Network with LiDAR Supervision for Autonomous Driving
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Institute of Control Engineering(北京控制工程研究所)
Software-Hardware Co-optimization for Modular E2E AV Paradigm: A Unified Framework of Optimization Approaches, Simulation Environment and Evaluation Metrics
School of Transportation, Southeast University(东南大学交通学院);School of Electronic Science and Engineering, Southeast University(东南大学电子科学与工程学院);Department of Civil and Systems Engineering, Johns Hopkins University(约翰霍普金斯大学土木与系统工程系)
TSLA: A Task-Specific Learning Adaptation for Semantic Segmentation on Autonomous Vehicles Platform
Department of Electrical and Computer Engineering, Northeastern University(电气与计算机工程系,东北大学);Robotics Institute, Carnegie Mellon University(机器人研究所,卡内基梅隆大学);School of Computing at the University of Georgia(佐治亚大学计算机学院);Knight Foundation School of Computing & Information Sciences at Florida International University(佛罗里达国际大学Knight基金会计算机与信息科学学院)
Weakly Supervised Multi-Modal 3D Human Body Pose Estimation for Autonomous Driving
University of Stuttgart(斯图加特大学);Friedrich-Alexander-Universität Erlangen-Nuernberg(埃尔朗根-纽伦堡弗里德里希-亚历山大大学);Ulm University(乌尔姆大学);Munich University of Applied Sciences(慕尼黑应用技术大学)