面向城市：车外参照的多模态解析
Speak to the City: Multimodal Resolution for Outside-the-Vehicle References
- Mercedes-Benz Tech Innovation GmbH（梅赛德斯-奔驰科技创新有限公司）
- Saarland University（萨尔大学）
- RheinMain University of Applied Sciences（莱茵美因应用科学大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对自动驾驶中车外参照的指代歧义问题，提出融合注视与语言的多模态框架，利用VR数据训练轻量Transformer，实现83.33%的Top-1准确率和24.3毫秒推理，支持实时空间检索。
AI中文摘要：
随着自动驾驶车辆和扩展现实（XR）头显实现新颖的车内交互，无缝查询物理地标（即车外参照，OVR）仍因自我运动和指代歧义而具有挑战性。我们提出了一种稳健的多模态OVR框架，融合用户注视和自然语言来识别兴趣点（POI）。为解决动态车辆数据的稀缺性，我们开发了基于VR的流水线，将360度公交视频与车辆GNSS遥测数据同步。通过一项用户研究（N=46），将乘客头部朝向映射到3D地理空间数字孪生中，我们捕获了真实的注视-语音行为。随后，我们训练了一个轻量级Transformer网络，利用大语言模型（LLM）将连续空间注视向量与离散语言上下文动态对齐。实验结果表明，该方法具有高准确率和低计算开销，实现了83.33%的Top-1准确率（87.72%的Top-2）和平均24.3毫秒的推理时间。这种实时范式有效解决了指代歧义，使车内乘客能够进行上下文感知的空间检索。
英文摘要：
As autonomous vehicles and Extended Reality (XR) headsets enable novel in-car interactions, seamlessly querying physical landmarks, known as Outside-the-Vehicle Referencing (OVR), remains challenging due to ego-motion and referential ambiguity. We present a robust, multimodal OVR framework fusing user gaze and natural language to identify Points of Interest (POIs). To address the scarcity of dynamic vehicular data, we developed a VR-based pipeline synchronizing 360-degree transit videos with vehicle GNSS telemetry. Through a user study (N=46) mapping passenger head orientation into a 3D geospatial Digital Twin, we captured authentic gaze-speech behaviors. We subsequently trained a lightweight Transformer network, leveraging LLMs to dynamically align continuous spatial gaze vectors with discrete verbal context. Experimental results demonstrate high accuracy and low computational overhead, achieving an 83.33% Top-1 accuracy (87.72% Top-2) and an average inference time of 24.3 milliseconds. This real-time paradigm effectively resolves referential ambiguity, enabling context-aware spatial retrieval for passengers within the vehicle.
