机器人何时应说'我不知道'：身体化问答中退避的基准测试
When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answering
AI总结：
本文研究了身体化问答中退避机制的重要性，通过构建AbstainEQA数据集，发现人类在退避任务中表现优异，而模型退避能力有限。
摘要：
身体化问答（EQA）要求一个智能体解读语言、感知其环境，并在3D场景中导航以产生响应。现有的EQA基准假设每个问题都必须回答，但身体化的智能体应知道何时缺乏足够的信息来回答。在本工作中，我们关注EQA智能体的最小要求，即退避：知道何时不给出回答。从对500个人类查询的初步研究中，我们发现32.4%的查询包含缺失或未明确说明的上下文。基于此初步研究和人类沟通错误的认知理论，我们推导出五个需要退避的代表性类别：行动限制、指代未明确说明、偏好依赖、信息不可用以及虚假预设。我们通过让标注者将规范问题转换为由这些类别概述的模糊变体来增强OpenEQA。所得到的数据集AbstainEQA包含1,636个标注的退避案例，与1,636个原始OpenEQA实例配对，以实现平衡评估。在AbstainEQA上评估时，我们发现即使是最前沿的模型也只能达到42.79%的退避召回率，而人类达到91.17%。我们还发现，扩展、提示和推理仅能带来微小的提升，而微调的模型会过度依赖文本提示。这些结果将退避定位为可靠交互在身体化环境中的基本前提，并作为有效澄清的必要基础。
英文摘要：
Embodied Question Answering (EQA) requires an agent to interpret language, perceive its environment, and navigate within 3D scenes to produce responses. Existing EQA benchmarks assume that every question must be answered, but embodied agents should know when they do not have sufficient information to answer. In this work, we focus on a minimal requirement for EQA agents, abstention: knowing when to withhold an answer. From an initial study of 500 human queries, we find that 32.4% contain missing or underspecified context. Drawing on this initial study and cognitive theories of human communication errors, we derive five representative categories requiring abstention: actionability limitation, referential underspecification, preference dependence, information unavailability, and false presupposition. We augment OpenEQA by having annotators transform well-posed questions into ambiguous variants outlined by these categories. The resulting dataset, AbstainEQA, comprises 1,636 annotated abstention cases paired with 1,636 original OpenEQA instances for balanced evaluation. Evaluating on AbstainEQA, we find that even the best frontier model only attains 42.79% abstention recall, while humans achieve 91.17%. We also find that scaling, prompting, and reasoning only yield marginal gains, and that fine-tuned models overfit to textual cues. Together, these results position abstention as a fundamental prerequisite for reliable interaction in embodied settings and as a necessary basis for effective clarification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
