arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07592cs.RO
收藏

从感知流中对大语言模型进行时空 grounding

Spatio-Temporal Grounding of Large Language Models from Perception Streams

Toyota Motor North America Research & Development(丰田北美研发中心)

Jacob Anderson, Bardh Hoxha, Georgios Fainekos, Hideki Okamoto, Danil Prokhorov

AI总结:

本文提出FESTS框架,通过将自然语言查询转化为SpRE,为大语言模型注入可验证的时空监督,提升时空推理能力,使30亿参数模型在27k数据上实现87.5%的帧级F1分数,接近GPT-4.1性能。

摘要:

具身AI代理必须推理物体在三维空间中随时间运动和交互的方式,但现有较小的前沿大语言模型(LLMs)仍无法正确处理细粒度的空间关系、度量距离和时间顺序。我们引入了通用框架Formally Explainable Spatio-Temporal Scenes(FESTS),通过将自然语言查询编译成空间正则表达式(SpRE)——一种结合正则表达式语法和S4u空间逻辑并扩展了全称和存在量词的语言,向LLM注入可验证的时空监督。该流程将每个SpRE与任何结构化的视频日志匹配,并导出对齐的(查询、帧、匹配、解释)元组,从而在无需人工标注的情况下实现无限训练数据。在27k这样的元组上训练一个30亿参数模型,将帧级F1从48.5%提升到87.5%,在复杂时空推理任务上接近GPT-4.1性能,同时比其小两个数量级,从而为视频LLM实现时空智能。

英文摘要:

Embodied-AI agents must reason about how objects move and interact in 3-D space over time, yet existing smaller frontier Large Language Models (LLMs) still mis-handle fine-grained spatial relations, metric distances, and temporal orderings. We introduce the general framework Formally Explainable Spatio-Temporal Scenes (FESTS) that injects verifiable spatio-temporal supervision into an LLM by compiling natural-language queries into Spatial Regular Expression (SpRE) -- a language combining regular expression syntax with S4u spatial logic and extended here with universal and existential quantification. The pipeline matches each SpRE against any structured video log and exports aligned (query, frames, match, explanation) tuples, enabling unlimited training data without manual labels. Training a 3-billion-parameter model on 27k such tuples boosts frame-level F1 from 48.5% to 87.5%, matching GPT-4.1 on complex spatio-temporal reasoning while remaining two orders of magnitude smaller, and, hence, enabling spatio-temporal intelligence for Video LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑