探索直至自信：面向具身问答的高效探索
Explore until Confident: Efficient Exploration for Embodied Question Answering
AI总结：
本文提出结合 VLM 视觉提示语义建图与保形预测置信度校准的 EQA 高效探索框架，并构建基于 HM3D 的新数据集，仿真和真实实验均验证其性能与效率提升。
摘要：
我们研究具身问答（Embodied Question Answering，EQA）问题，它指的是机器人等具身智能体需要主动探索环境以收集信息，直到其对某个问题的答案感到自信的场景。在本工作中，我们利用大型视觉-语言模型（VLM）强大的语义推理能力来高效探索并回答此类问题。然而，在 EQA 中使用 VLM 存在两个主要挑战：它们没有用于构建场景地图的内部记忆，因而无法规划如何随时间推进探索；同时，其置信度可能校准不当，导致机器人过早停止探索或过度探索。我们提出一种方法：首先基于深度信息并通过对 VLM 进行视觉提示来构建场景的语义地图——利用其关于场景中相关区域的丰富知识来指导探索。接着，我们使用保形预测来校准 VLM 的问答置信度，使机器人知道何时停止探索，从而得到校准更好且更高效的探索策略。为在仿真中测试我们的框架，我们还基于 Habitat-Matterport 3D Research Dataset（HM3D）贡献了一个新的 EQA 数据集，其中包含多样、真实的人机场景。仿真和真实机器人实验均表明，与未利用 VLM 进行探索或未校准其置信度的基线相比，我们提出的方法提升了性能和效率。包含实验视频和代码的网页：https://explore-eqa.github.io/
英文摘要：
We consider the problem of Embodied Question Answering (EQA), which refers to settings where an embodied agent such as a robot needs to actively explore an environment to gather information until it is confident about the answer to a question. In this work, we leverage the strong semantic reasoning capabilities of large vision-language models (VLMs) to efficiently explore and answer such questions. However, there are two main challenges when using VLMs in EQA: they do not have an internal memory for mapping the scene to be able to plan how to explore over time, and their confidence can be miscalibrated and can cause the robot to prematurely stop exploration or over-explore. We propose a method that first builds a semantic map of the scene based on depth information and via visual prompting of a VLM - leveraging its vast knowledge of relevant regions of the scene for exploration. Next, we use conformal prediction to calibrate the VLM's question answering confidence, allowing the robot to know when to stop exploration - leading to a more calibrated and efficient exploration strategy. To test our framework in simulation, we also contribute a new EQA dataset with diverse, realistic human-robot scenarios and scenes built upon the Habitat-Matterport 3D Research Dataset (HM3D). Both simulated and real robot experiments show our proposed approach improves the performance and efficiency over baselines that do no leverage VLM for exploration or do not calibrate its confidence. Webpage with experiment videos and code: https://explore-eqa.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
