观察与聆听内外：多模态人工智能系统用于驾驶员安全评估和智能车辆决策
Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making
AI总结：
本文提出L-LIO框架，通过融合音频与视觉数据提升驾驶员状态评估和环境理解，探讨音频在车辆安全中的作用，包括驾驶员语音分类、乘客指令分析及视觉不足时的音频辅助。
摘要：
looking-in-looking-out（LILO）框架已使智能车辆应用能够理解外部场景和驾驶员状态以提升安全性能，例如智能安全气囊部署、自动驾驶过渡中的接管时间预测以及驾驶员注意力监控。本文提出对LILO框架的扩展，论证音频模态作为理解驾驶员的额外信息源，并在自动驾驶发展背景下也涵盖乘客和外部人员。通过整合音频信号，扩展LILO为looking-and-listening inside-and-outside（L-LIO）框架，通过多模态传感器融合增强驾驶员状态评估和环境理解。评估了三个案例：利用驾驶员语音音频分类潜在障碍状态（如醉酒）、分析乘客自然语言指令（如“在那红色建筑后转弯”）以展示语音如何通过音频对齐指令数据与规划系统交互，以及视觉单一系统在指导和手势歧义时音频的辅助作用。数据集包含定制收集的车内和外部真实环境音频样本。初步发现显示音频在安全相关洞察方面有贡献，特别是在复杂或情境丰富的场景中，声音对安全决策至关重要或视觉信号不足时。挑战包括环境噪声干扰、隐私考虑和跨人类主体的鲁棒性，推动进一步研究动态真实世界中的可靠性。L-LIO通过音频和视觉传感的多模态融合增强驾驶员和场景理解，提供新的安全干预路径。
英文摘要：
The looking-in-looking-out (LILO) framework has enabled intelligent vehicle applications that understand both the outside scene and the driver state to improve safety outcomes, with examples in smart airbag deployment, takeover time prediction in autonomous control transitions, and driver attention monitoring. In this research, we propose an augmentation to this framework, making a case for the audio modality as an additional source of information to understand the driver, and in the evolving autonomy landscape, also the passengers and those outside the vehicle. We expand LILO by incorporating audio signals, forming the looking-and-listening inside-and-outside (L-LIO) framework to enhance driver state assessment and environment understanding through multimodal sensor fusion. We evaluate three example cases where audio enhances vehicle safety: supervised learning on driver speech audio to classify potential impairment states (e.g., intoxication), collection and analysis of passenger natural language instructions (e.g., "turn after that red building") to motivate how spoken language can interface with planning systems through audio-aligned instruction data, and limitations of vision-only systems where audio may disambiguate the guidance and gestures of external agents. Datasets include custom-collected in-vehicle and external audio samples in real-world environments. Pilot findings show that audio yields safety-relevant insights, particularly in nuanced or context-rich scenarios where sound is critical to safe decision-making or visual signals alone are insufficient. Challenges include ambient noise interference, privacy considerations, and robustness across human subjects, motivating further work on reliability in dynamic real-world contexts. L-LIO augments driver and scene understanding through multimodal fusion of audio and visual sensing, offering new paths for safety intervention.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
