arXiv 2606.081232026-08-17cs.CVcs.AI

以人为中心的驾驶员监控模型基准测试

Beyond aggregate scores: Deployment-aware and non-compensatory benchmarking of vision-based eye-state recognition models for driver monitoring

Universidad Nacional de San Agustin de Arequipa (UNSA)（圣奥古斯丁国立大学（UNSA））

Ruben Dario Florez-Zela

备注：: 32 pages total (19 main + 13 supplementary). v2 substantially extends v1 from four to six architectures; adds RT-BENE zero-shot and participant-safe target-domain OOF evaluation, deterministic corruptions, Jetson Nano TensorRT benchmarking, RISE faithfulness/stability, statistical inference, and non-compensatory eligibility. Code: https://github.com/rubendflorezzela/hcbf-driver-monitoring

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对驾驶员监控模型仅用分类精度评估的不足，提出以人为中心的基准测试框架（HCBF），从精度、可解释性、效率和鲁棒性四维评估，发现模型在帕累托前沿上各占优势，但聚合排名会掩盖关键缺陷。

摘要：

基于视觉的驾驶员监控系统越来越多地部署在安全关键的智能交通环境中，但它们几乎总是仅根据分类精度进行比较。本文认为精度不足以表征模型在实际部署中的适用性，并提出了以人为中心的基准测试框架（HCBF），该框架从四个维度评估模型：精度、可解释性、效率和鲁棒性。该框架应用于四种代表性的轻量级架构：MobileNetV3、ShuffleNetV2、EfficientNet-B0和DeiT-Tiny，在MRL眼睛数据集上进行眼睛状态分类。虽然这些模型在干净数据集上的精度几乎无法区分，但每个模型恰好在一个维度上领先，并且所有四个模型都位于帕累托前沿。在三种面向部署的权重场景下计算的人为中心得分将ShuffleNetV2排在首位。然而，这个聚合胜出者在传感器噪声下保留了不到一半的性能，并且将闭眼分类为睁眼而失败，而Transformer则保持鲁棒。这些发现表明，聚合排名可能掩盖在操作上具有决定性的维度特定漏洞，强调了多维、以人为中心评估的价值。

英文摘要：

Model selection for safety-relevant visual recognition is often based on clean aggregate performance, although robustness, transfer, embedded latency, and explanation faithfulness may produce different preferences. This study presents a Human-Centered Benchmarking Framework (HCBF) that separates multidimensional evidence from non-compensatory operational eligibility. Six compact convolutional and transformer-oriented eye-state recognition models were evaluated using a subject-disjoint MRL Eye protocol, deterministic image corruptions, zero-shot transfer and participant-safe target-domain training with out-of-fold evaluation on RT-BENE, TensorRT FP32 inference on an NVIDIA Jetson Nano, and black-box RISE faithfulness. Clean MRL Macro-F1 ranged from 0.9566 to 0.9794, whereas zero-shot RT-BENE Macro-F1 ranged from 0.2066 to 0.7771. Matched target-domain effects varied from -0.0406 to 0.4807 and redistributed the two directional errors differently across architectures. Only MobileNetV3-Large and ShuffleNetV2 met the 33.333-ms binocular-pair latency deadline, while neither passed the predefined safety-related screen. The remaining four models failed both requirements, yielding an empty eligible set. Normalized deletion AUC ranged from 0.5826 to 0.9113, while normalized insertion coverage varied from 17.6% to 88.6%. Model ordering changed across clean prediction, corruption robustness, transfer, deployment, faithfulness, and historical score sensitivity. These findings show that relative ranking, multidimensional preference, and operational eligibility are distinct decisions. Deployment-aware benchmarking should preserve directional failures and uncertainty and should allow no model to be selected when mandatory requirements are unmet.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。