arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03993cs.HC
收藏

TR-LLM:融合轨迹数据用于场景感知的基于大语言模型的人类行为预测

TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction

Kojiro Takeyama, Yimeng Liu, Misha Sra

AI总结:

本文提出TR-LLM多模态框架,通过融合人类轨迹数据中的物理约束来增强大语言模型的空间感知能力,实验证明该方法在场景信息受限时能显著提升人类行为预测性能。

摘要:

准确预测人类行为对于AI系统有效支持现实世界应用至关重要,例如自主机器人预测并协助人类任务。现实场景经常面临遮挡和不完整的场景观测等挑战,这可能损害预测准确性。因此,传统的基于视频的方法往往因时间和空间视角有限而表现不佳。大语言模型(LLMs)提供了一种有前景的替代方案。由于在描述人类行为的大型文本语料库上进行了训练,LLMs很可能编码了家庭环境中合理的人类动作序列。然而,主要基于文本数据训练的LLMs缺乏固有的空间意识和实时环境感知能力。它们在理解物理约束和空间几何方面存在困难。因此,为了在现实世界空间场景中有效发挥作用,我们提出了一种多模态预测框架,通过融合从人类轨迹中推导出的物理约束来增强基于LLM的行为预测。我们的实验表明,将LLM预测与轨迹数据相结合显著提高了整体预测性能。这种提升在LLM接收到的场景信息有限的情况下尤为显著,凸显了语言知识与物理约束在理解和预测人类行为方面的互补性。

英文摘要:

Accurate prediction of human behavior is crucial for AI systems to effectively support real-world applications, such as autonomous robots anticipating and assisting with human tasks. Real-world scenarios frequently present challenges such as occlusions and incomplete scene observations, which can compromise predictive accuracy. Thus, traditional video-based methods often struggle due to limited temporal and spatial perspectives. Large Language Models (LLMs) offer a promising alternative. Having been trained on a large text corpus describing human behaviors, LLMs likely encode plausible sequences of human actions in a home environment. However, LLMs, trained primarily on text data, lack inherent spatial awareness and real-time environmental perception. They struggle with understanding physical constraints and spatial geometry. Therefore, to be effective in a real-world spatial scenario, we propose a multimodal prediction framework that enhances LLM-based action prediction by integrating physical constraints derived from human trajectories. Our experiments demonstrate that combining LLM predictions with trajectory data significantly improves overall prediction performance. This enhancement is particularly notable in situations where the LLM receives limited scene information, highlighting the complementary nature of linguistic knowledge and physical constraints in understanding and anticipating human behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑