arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.02517cs.CV
收藏

HENet:面向多视角相机端到端多任务3D感知的混合编码方法

HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from Multi-view Cameras

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Chongqing Changan Automobile Co., Ltd.(重庆长安汽车股份有限公司);University of California, Merced(加州大学默塞德分校)

Zhongyu Xia, ZhiWei Lin, Xinhao Wang, Yongtao Wang, Yun Xing, Shengxiang Qi, Nan Dong, Ming-Hsuan Yang

AI总结:

针对多任务3D感知中计算资源受限、任务冲突等问题,提出混合编码的端到端框架HENet,结合大小编码器处理长短时序帧,在nuScenes基准上取得SOTA结果。

摘要:

多视角相机的三维感知是自动驾驶系统的关键组成部分,涉及3D目标检测、鸟瞰图(BEV)语义分割等多项任务。为提升感知精度,近期的3D感知模型采用了大型图像编码器、高分辨率图像以及长时序输入,带来了显著的性能提升。然而,受计算资源限制,这些技术在训练和推理场景中往往难以兼容。此外,现代自动驾驶系统更倾向于采用端到端框架实现多任务3D感知,此举可简化整体系统架构并降低实现复杂度,但在端到端3D感知模型中联合优化多个任务时,常会出现任务间的冲突问题。为缓解上述问题,本文提出了一种面向多任务3D感知的端到端框架HENet。具体而言,我们提出了一种混合图像编码网络,对短期帧采用大型图像编码器,对长期时序帧采用小型图像编码器。随后,我们引入了基于注意力机制的时序特征整合模块,用于融合上述两个混合图像编码器提取的不同帧的特征。最后,我们根据各感知任务的特点,为不同任务采用不同网格尺寸的BEV特征、独立的BEV编码器以及任务解码器。实验结果表明,HENet在nuScenes基准上取得了当前最优的端到端多任务3D感知结果,涵盖3D目标检测和BEV语义分割任务。源代码和模型将发布于https://github.com/VDIGPKU/HENet。

英文摘要:

Three-dimensional perception from multi-view cameras is a crucial component in autonomous driving systems, which involves multiple tasks like 3D object detection and bird's-eye-view (BEV) semantic segmentation. To improve perception precision, large image encoders, high-resolution images, and long-term temporal inputs have been adopted in recent 3D perception models, bringing remarkable performance gains. However, these techniques are often incompatible in training and inference scenarios due to computational resource constraints. Besides, modern autonomous driving systems prefer to adopt an end-to-end framework for multi-task 3D perception, which can simplify the overall system architecture and reduce the implementation complexity. However, conflict between tasks often arises when optimizing multiple tasks jointly within an end-to-end 3D perception model. To alleviate these issues, we present an end-to-end framework named HENet for multi-task 3D perception in this paper. Specifically, we propose a hybrid image encoding network, using a large image encoder for short-term frames and a small image encoder for long-term temporal frames. Then, we introduce a temporal feature integration module based on the attention mechanism to fuse the features of different frames extracted by the two aforementioned hybrid image encoders. Finally, according to the characteristics of each perception task, we utilize BEV features of different grid sizes, independent BEV encoders, and task decoders for different tasks. Experimental results show that HENet achieves state-of-the-art end-to-end multi-task 3D perception results on the nuScenes benchmark, including 3D object detection and BEV semantic segmentation. The source code and models will be released at https://github.com/VDIGPKU/HENet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑