HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from Multi-view Cameras
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Chongqing Changan Automobile Co., Ltd.(重庆长安汽车股份有限公司);University of California, Merced(加州大学默塞德分校)
Zhongyu Xia, ZhiWei Lin, Xinhao Wang, Yongtao Wang, Yun Xing, Shengxiang Qi, Nan Dong, Ming-Hsuan Yang
Three-dimensional perception from multi-view cameras is a crucial component in autonomous driving systems, which involves multiple tasks like 3D object detection and bird's-eye-view (BEV) semantic segmentation. To improve perception precision, large image encoders, high-resolution images, and long-term temporal inputs have been adopted in recent 3D perception models, bringing remarkable performance gains. However, these techniques are often incompatible in training and inference scenarios due to computational resource constraints. Besides, modern autonomous driving systems prefer to adopt an end-to-end framework for multi-task 3D perception, which can simplify the overall system architecture and reduce the implementation complexity. However, conflict between tasks often arises when optimizing multiple tasks jointly within an end-to-end 3D perception model. To alleviate these issues, we present an end-to-end framework named HENet for multi-task 3D perception in this paper. Specifically, we propose a hybrid image encoding network, using a large image encoder for short-term frames and a small image encoder for long-term temporal frames. Then, we introduce a temporal feature integration module based on the attention mechanism to fuse the features of different frames extracted by the two aforementioned hybrid image encoders. Finally, according to the characteristics of each perception task, we utilize BEV features of different grid sizes, independent BEV encoders, and task decoders for different tasks. Experimental results show that HENet achieves state-of-the-art end-to-end multi-task 3D perception results on the nuScenes benchmark, including 3D object detection and BEV semantic segmentation. The source code and models will be released at https://github.com/VDIGPKU/HENet.
Unveiling the Black Box: Independent Functional Module Evaluation for Bird's-Eye-View Perception Model
School of Vehicle and Mobility, Tsinghua University(清华大学车辆与学院);State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University(清华大学智能绿色车辆与交通全国重点实验室);School of Mathematical Sciences, Nankai University(南开大学数学科学学院);School of Electronic and Information Engineering, Beijing University of Aeronautics and Astronautics(北京航空航天大学电子信息工程学院)
Sensor Fusion by Spatial Encoding for Autonomous Driving
Cho Chun Shik Graduate School of Mobility(赵春植 mobility 研究生院(注:该学院为KAIST下属学院,常见译法为赵春植移动研究生院));Division of Future Vehicle(未来车辆学部);Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST))
ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision-and-Language Navigation in Continuous Environments
Robotics and Control Laboratory, the School of Advanced Manufacturing and Robotics, and the State Key Laboratory of Turbulence and Complex Systems, Peking University(机器人与控制实验室、先进制造与机器人学院、湍流与复杂系统国家重点实验室,北京大学);Defense Innovation Institute, Academy of Military Sciences(国防创新研究院,军事科学院);Tianjin Artificial Intelligence Innovation Center(天津人工智能创新中心);Harbin Institute of Technology(哈尔滨工业大学)
Look, Zoom, Understand: The Robotic Eyeball for Embodied Perception
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Dalian University of Technology(大连理工大学)
Toward a Low-Cost Perception System in Autonomous Vehicles: A Spectrum Learning Approach
Department of Electrical and Computer Engineering, College of Engineering(电气与计算机工程系,工程学院);The Robotics Institute, School of Computer Science(机器人研究所,计算机科学学院)
LSVG: Language-Guided Scene Graphs with 2D-Assisted Multi-Modal Encoding for 3D Visual Grounding
School of Automation Science and Engineering, South China University of Technology(自动化科学与工程学院,华南理工大学);School of Future Technology, South China University of Technology(未来技术学院,华南理工大学)
TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception
The University of Hong Kong(香港大学);Yale University(耶鲁大学);Shanghai AI Laboratory(上海人工智能实验室);HKU Shanghai Intelligent Computing Research Center(香港大学上海智能计算研究中心)
A Joint Visual Compression and Perception Framework for Neuralmorphic Spiking Camera
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);National Engineering Research Center of Visual Technology, Peking University(北京大学视觉技术国家工程研究中心);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究所)
RETR: Multi-View Radar Detection Transformer for Indoor Perception
Mitsubishi Electric Research Laboratories (MERL)(三菱电机研究实验室(MERL));University of Bologna(博洛尼亚大学);Information Technology R&D Center (ITC), Mitsubishi Electric Corporation(三菱电机公司信息技术研发中心(ITC))
3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
School of Geospatial Engineering and Science(测绘工程与科学学院);Sun Yat-sen University(中山大学);College of Electronic Science(电子科学学院);National University of Defense Technology(国防科技大学);Department of Civil and Environmental Engineering(土木与环境工程系);Norwegian University of Science and Technology(挪威科技大学);National Geomatics Center of China(中国国家测绘地理信息中心)