Semantic Flip: 用于具身问答和空间定位中鲁棒拒绝的合成OOD生成
Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localization
AI总结：
提出Semantic Flip框架，通过合成辅助OOD样本训练轻量拒绝模块，使冻结的视觉语言模型在无外部OOD标注下实现鲁棒拒绝，在具身问答和空间定位基准上优于强提示基线。
摘要：
检测不可回答的用户查询对于现实世界具身代理的可靠部署仍然至关重要。然而，现代视觉语言模型（VLM）即使当可用视觉记忆无法支持查询时，也常常生成过于自信的答案。这种过度自信会带来各种任务依赖的风险。代理可能在具身问答中向用户提供误导信息，并在空间推理导航中选择任意坐标并物理引导用户前往。尽管风险很高，但只有少数先前研究直接解决具身VLM何时以及如何回答“我不知道”的问题。本文提出Semantic Flip，一个简单而有效的框架，无需外部OOD标注即可合成辅助分布外（OOD）样本用于具身拒绝。关键思想是独立变换查询和视频记忆，以构建缺乏足够视觉基础的辅助OOD对。这些合成对使得能够在冻结的预训练VLM之上训练一个轻量级拒绝模块。该模块可附加到任何现有的基于VLM的流水线中，无需重新训练底层模型。在两个互补的基准测试中，Semantic Flip始终优于强提示基线。本文还引入了SpaceReject，一个新的用于空间定位的拒绝基准，包含故意不可回答的查询和长视频记忆，其中Semantic Flip达到了0.9559的$F_1$分数。源代码和数据集公开于https://github.com/ndb796/SemanticFlip。
英文摘要：
Detecting unanswerable user queries remains essential for the reliable deployment of real-world embodied agents. However, modern vision-language models (VLMs) often generate overly confident answers even when the available visual memory cannot support the query. Such overconfidence poses various task-dependent risks. The agent may provide misleading information to the user in Embodied Question Answering and select an arbitrary coordinate and physically guide the user there in spatial reasoning for navigation. Despite these high stakes, only a few prior studies directly address when and how an embodied VLM should respond with "I do not know." This work proposes Semantic Flip, a simple yet effective framework that synthesizes auxiliary out-of-distribution (OOD) samples for embodied refusal without requiring external OOD annotations. The key idea is to independently transform the query and video memory to construct auxiliary OOD pairs that lack sufficient visual grounding. These synthesized pairs enable training a lightweight rejection module on top of a frozen pretrained VLM. The module attaches to any existing VLM-based pipeline without retraining the underlying model. Across two complementary benchmarks, Semantic Flip consistently outperforms strong prompting baselines. This work also introduces SpaceReject, a new refusal benchmark for spatial localization with deliberately unanswerable queries over long video memory, where Semantic Flip achieves an $F_1$ score of 0.9559. The source codes and datasets are publicly available at https://github.com/ndb796/SemanticFlip.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
