t-READi:面向自动驾驶的Transformer驱动的鲁棒高效多模态推理
t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous Driving
AI总结:
t-READi提出一种自适应多模态推理系统,通过仅调整对变化敏感的特定参数并利用跨模态对比学习补偿缺失模态,在真实传感器变化条件下实现精度提升6%以上、延迟降低近15倍。
摘要:
鉴于自动驾驶车辆(AV)广泛采用多模态传感器(如摄像头、激光雷达、雷达),对这些传感器输出进行深度融合以实现鲁棒感知变得至关重要。然而,现有融合方法通常依赖两个在实践中很少成立的假设:i)所有输入具有相似的数据分布;ii)所有传感器始终可用。例如,由于激光雷达具有不同的分辨率,且雷达可能发生故障,这种可变性往往导致融合性能显著下降。为此,我们提出了t-READi,一种自适应推理系统,能够适应多模态传感数据的可变性,从而实现鲁棒且高效的感知。t-READi识别对变化敏感但结构特定的模型参数;然后仅自适应这些参数,同时保持其余部分不变。t-READi还利用跨模态对比学习方法,以补偿缺失模态造成的损失。这两项功能的实现均保持与现有多模态深度融合方法的兼容性。大量实验清楚地表明,与现有方法相比,在真实数据和模态变化条件下,t-READi不仅将平均推理精度提高了6%以上,而且将推理延迟降低了近15倍,即使在最坏情况下也仅增加5%的额外内存开销。
英文摘要:
Given the wide adoption of multimodal sensors (e.g., camera, lidar, radar) by autonomous vehicles (AVs), deep analytics to fuse their outputs for a robust perception become imperative. However, existing fusion methods often make two assumptions rarely holding in practice: i) similar data distributions for all inputs and ii) constant availability for all sensors. Because, for example, lidars have various resolutions and failures of radars may occur, such variability often results in significant performance degradation in fusion. To this end, we present tREADi, an adaptive inference system that accommodates the variability of multimodal sensory data and thus enables robust and efficient perception. t-READi identifies variation-sensitive yet structure-specific model parameters; it then adapts only these parameters while keeping the rest intact. t-READi also leverages a cross-modality contrastive learning method to compensate for the loss from missing modalities. Both functions are implemented to maintain compatibility with existing multimodal deep fusion methods. The extensive experiments evidently demonstrate that compared with the status quo approaches, t-READi not only improves the average inference accuracy by more than 6% but also reduces the inference latency by almost 15x with the cost of only 5% extra memory overhead in the worst case under realistic data and modal variations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。