感知、反思与规划:设计无需指令的目标导向城市导航 LLM 智能体
Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions
AI总结:
本文提出一种感知-反思-规划工作流,通过微调 LLaVA-7B 感知地标方位距离、引入记忆机制进行反思并生成长期计划,显著提升 LLM 智能体在无指令城市导航中的长距离决策能力。
摘要:
本文考虑城市导航中的一个场景:为 AI 智能体提供目标位置相对于某些知名地标的语言描述;智能体仅通过观察周围场景,包括识别地标和道路网络连接,就必须在没有任何指令的情况下做出决策并导航到目标位置。该问题极具挑战性,因为它要求智能体建立自身定位并获取复杂城市环境的空间表征,而地标往往不可见。在缺乏导航指令的情况下,这些能力对于智能体在长距离城市导航中做出高质量决策至关重要。随着大语言模型(LLMs)涌现出的推理能力,一个诱人的基线方法是提示 LLM 对每次观察进行“反应”并据此做出决策。然而,该基线方法表现非常差,智能体经常反复访问相同位置,并做出短视且不一致的决策。为了解决这些问题,本文引入了一种新颖的智能体工作流,其特点是具备感知、反思和规划能力。具体而言,我们发现 LLaVA-7B 可以通过微调来感知地标的方向和距离,其精度足以支持城市导航。此外,反思通过一种记忆机制实现,过去的经验被存储起来,并可以与当前感知一起检索,用于有效的决策论证。规划利用反思结果生成长期计划,从而避免长距离导航中的短视决策。我们表明,与最先进的基线方法相比,所设计的工作流显著提升了 LLM 智能体的导航能力。
英文摘要:
This paper considers a scenario in city navigation: an AI agent is provided with language descriptions of the goal location with respect to some well-known landmarks; By only observing the scene around, including recognizing landmarks and road network connections, the agent has to make decisions to navigate to the goal location without instructions. This problem is very challenging, because it requires agent to establish self-position and acquire spatial representation of complex urban environment, where landmarks are often invisible. In the absence of navigation instructions, such abilities are vital for the agent to make high-quality decisions in long-range city navigation. With the emergent reasoning ability of large language models (LLMs), a tempting baseline is to prompt LLMs to "react" on each observation and make decisions accordingly. However, this baseline has very poor performance that the agent often repeatedly visits same locations and make short-sighted, inconsistent decisions. To address these issues, this paper introduces a novel agentic workflow featured by its abilities to perceive, reflect and plan. Specifically, we find LLaVA-7B can be fine-tuned to perceive the direction and distance of landmarks with sufficient accuracy for city navigation. Moreover, reflection is achieved through a memory mechanism, where past experiences are stored and can be retrieved with current perception for effective decision argumentation. Planning uses reflection results to produce long-term plans, which can avoid short-sighted decisions in long-range navigation. We show the designed workflow significantly improves navigation ability of the LLM agent compared with the state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。