基于掩码多头自注意力的鲁棒多视角多模态驾驶员监控系统
Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention
AI总结：
提出基于多头自注意力特征级融合的多视角多模态驾驶员监控系统，通过对比实验和掩码训练提升鲁棒性，在DAD数据集上达到97.0%的AUC-ROC。
摘要：
驾驶员监控系统（DMS）对于L2+级自动驾驶车辆中的安全交接操作至关重要。最先进的DMS利用安装在多个不同位置的传感器来监控驾驶员和车辆内部场景，并采用决策级融合来整合这些异构数据。然而，这种融合方法可能无法充分利用不同数据源之间的互补性，并且可能忽视它们的相对重要性。为了解决这些局限性，我们提出了一种新颖的基于多头自注意力（MHSA）特征级融合的多视角多模态驾驶员监控系统。我们通过将其与四种替代融合策略（Sum、Conv、SE和AFF）进行比较来证明其有效性。我们还提出了一种新颖的GPU友好的监督对比学习框架SuMoCo，以学习更好的表示。此外，我们对DAD数据集的测试集进行了细粒度划分，以实现驾驶员活动的多类识别。在这个增强数据库上的实验表明：1）所提出的基于MHSA的融合方法（AUC-ROC：97.0%）优于所有基线和先前的方法；2）使用补丁掩码训练MHSA可以提高其对模态/视角崩溃的鲁棒性。代码和标注已公开提供。
英文摘要：
Driver Monitoring Systems (DMSs) are crucial for safe hand-over actions in Level-2+ self-driving vehicles. State-of-the-art DMSs leverage multiple sensors mounted at different locations to monitor the driver and the vehicle's interior scene and employ decision-level fusion to integrate these heterogenous data. However, this fusion method may not fully utilize the complementarity of different data sources and may overlook their relative importance. To address these limitations, we propose a novel multiview multimodal driver monitoring system based on feature-level fusion through multi-head self-attention (MHSA). We demonstrate its effectiveness by comparing it against four alternative fusion strategies (Sum, Conv, SE, and AFF). We also present a novel GPU-friendly supervised contrastive learning framework SuMoCo to learn better representations. Furthermore, We fine-grained the test split of the DAD dataset to enable the multi-class recognition of drivers' activities. Experiments on this enhanced database demonstrate that 1) the proposed MHSA-based fusion method (AUC-ROC: 97.0\%) outperforms all baselines and previous approaches, and 2) training MHSA with patch masking can improve its robustness against modality/view collapses. The code and annotations are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
