PerspAct：通过视角采择与主动视觉增强大语言模型的情境协作能力
PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision
AI总结：
本研究针对LLM在多智能体协作中视角采择能力不足的问题，基于ReAct框架扩展经典Director任务设计七个场景，通过主动视觉探索增强视角理解，显著提升模型协作有效性，为情境感知AI系统研究奠基。
摘要：
大语言模型（LLMs）和多模态基础模型的最新进展显著拓宽了其在机器人学和协作系统中的应用。然而，有效的多智能体交互需要强大的视角采择能力，使模型能够解释物理和认知视角。当前训练范式常忽视这些交互语境，导致模型在推理个体视角的主观性或在多观察者环境中导航时面临挑战。本研究评估使用ReAct框架（一种整合推理与行动的方法）明确纳入多样视角，能否增强LLM理解并落地其他智能体需求的能力。我们扩展经典Director任务，在七个视角采择复杂度递增的场景中引入主动视觉探索。这些场景旨在挑战智能体基于视觉访问和交互解决指称歧义的能力，涉及不同状态表示和提示策略（包括ReAct风格推理）。结果表明，明确的视角线索结合主动探索策略，显著提升模型的解释准确性和协作有效性。这些发现凸显了将主动感知与视角采择机制整合，在推进LLMs在机器人学和多智能体系统中应用的潜力，为未来自适应和情境感知AI系统的研究奠定基础。
英文摘要：
Recent advances in Large Language Models (LLMs) and multimodal foundation models have significantly broadened their application in robotics and collaborative systems. However, effective multi-agent interaction necessitates robust perspective-taking capabilities, enabling models to interpret both physical and epistemic viewpoints. Current training paradigms often neglect these interactive contexts, resulting in challenges when models must reason about the subjectivity of individual perspectives or navigate environments with multiple observers. This study evaluates whether explicitly incorporating diverse points of view using the ReAct framework, an approach that integrates reasoning and acting, can enhance an LLM's ability to understand and ground the demands of other agents. We extend the classic Director task by introducing active visual exploration across a suite of seven scenarios of increasing perspective-taking complexity. These scenarios are designed to challenge the agent's capacity to resolve referential ambiguity based on visual access and interaction, under varying state representations and prompting strategies, including ReAct-style reasoning. Our results demonstrate that explicit perspective cues, combined with active exploration strategies, significantly improve the model's interpretative accuracy and collaborative effectiveness. These findings highlight the potential of integrating active perception with perspective-taking mechanisms in advancing LLMs' application in robotics and multi-agent systems, setting a foundation for future research into adaptive and context-aware AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
