CAVEN：一种用于嘈杂环境中高效视听导航的具身对话智能体
CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
AI总结：
本文提出CAVEN框架，通过对话式交互和预算感知决策，在嘈杂环境中实现高效视听导航，显著提升成功率。
摘要：
智能体在视听导航中定位音频目标是一项具有挑战性的任务，尤其是在音频是零星的或环境嘈杂的情况下。在本文中，我们提出了CAVEN，一种基于对话的视听具身导航框架，其中智能体可以与人类/预言者交互以解决导航到音频目标的任务。具体来说，CAVEN被建模为一个预算感知的部分可观测半马尔可夫决策过程，该过程隐式地学习基于音频的导航策略中的不确定性，以决定智能体何时以及如何与预言者交互。我们的CAVEN智能体可以通过产生相关问题进行完全双向的自然语言对话，并根据视听上下文解释来自预言者的自由形式、可能嘈杂的响应。为了实现这种能力，CAVEN配备了：（i）一个轨迹预测网络，该网络以视听线索为基础，生成到估计目标的潜在轨迹，以及（ii）一个基于自然语言的问题生成和推理网络，用于向预言者提出交互式问题或解释预言者的响应以生成导航指令。为了训练交互模块，我们提出了一个大规模数据集：AVN-Instruct，基于Landmark-RxR数据集。为了证实对话的有用性，我们在基准音频目标任务上使用SoundSpaces模拟器在各种嘈杂设置下进行了实验。我们的结果表明，我们完全对话式的方法在成功率上带来了近一个数量级的提升，特别是在定位新声源以及对比仅使用单向交互的方法时。
英文摘要：
Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle. Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context. To enable such a capability, CAVEN is equipped with: (i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions. To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset. To substantiate the usefulness of conversations, we present experiments on the benchmark audio-goal task using the SoundSpaces simulator under various noisy settings. Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that only use uni-directional interaction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
