可访问的指令跟随智能体
Accessible Instruction-Following Agent
AI总结：
本研究提出UVLN框架，结合GPT3与BLIP，通过机器翻译增强跨语言视觉-语言导航，提升非英语指令跟随智能体的可交互性与可访问性。
摘要：
人类可以基于环境中的视觉信号和指令进行协作并完成任务。训练这样的机器人是困难的，尤其是由于对指令和复杂环境的理解。以往的指令跟随智能体偏向于以英语为中心的语料库，使其无法应用于使用多种语言甚至低资源语言的用户。然而，指令跟随智能体以假设用户能观察环境的模式进行预训练，这限制了其可访问性。在这项工作中，我们试图将指令跟随智能体的成功推广到语料库资源稀少的非英语语言，并提高其可交互性和可访问性。我们引入了UVLN（通用视觉-语言导航），一种新颖的用于跨语言视觉-语言导航的机器翻译指令增强框架，其创新性地组合了最先进的大语言模型（GPT3）与图像描述模型（BLIP）。我们首先通过机器翻译收集了一个多语言视觉-语言导航数据集。然后，我们通过跨语言编码器将标准的VLN训练目标扩展到多语言设置。不同语言之间的对齐通过共享的视觉和动作上下文，经由跨模态变换器捕获，该变换器编码语言指令、视觉观察和动作决策序列的输入。为了提高可交互性，我们将我们的智能体与大语言模型连接，该模型向用户告知情境和当前状态，并解释动作决策。在Room Across Room数据集上的实验证明了我们方法的有效性。定性结果显示了我们的指令跟随智能体具有良好的可交互性和可访问性。
英文摘要：
Humans can collaborate and complete tasks based on visual signals and instruction from the environment. Training such a robot is difficult especially due to the understanding of the instruction and the complicated environment. Previous instruction-following agents are biased to English-centric corpus, making it unrealizable to be applied to users that use multiple languages or even low-resource languages. Nevertheless, the instruction-following agents are pre-trained in a mode that assumes the user can observe the environment, which limits its accessibility. In this work, we're trying to generalize the success of instruction-following agents to non-English languages with little corpus resources, and improve its intractability and accessibility. We introduce UVLN (Universal Vision-Language Navigation), a novel machine-translation instructional augmented framework for cross-lingual vision-language navigation, with a novel composition of state-of-the-art large language model (GPT3) with the image caption model (BLIP). We first collect a multilanguage vision-language navigation dataset via machine translation. Then we extend the standard VLN training objectives to a multilingual setting via a cross-lingual language encoder. The alignment between different languages is captured through a shared vision and action context via a cross-modal transformer, which encodes the inputs of language instruction, visual observation, and action decision sequences. To improve the intractability, we connect our agent with the large language model that informs the situation and current state to the user and also explains the action decisions. Experiments over Room Across Room Dataset prove the effectiveness of our approach. And the qualitative results show the promising intractability and accessibility of our instruction-following agent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
