在紧凑世界模型中建立空间关系:指令泄漏与无目标动力学修复
Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix
AI总结:
研究以语言目标为条件的紧凑世界模型中空间关系建立问题,发现指令泄漏陷阱,提出将目标排除在动力学外并监督读取路径的修复方法,适用于相关世界模型,提升关系基础的准确性。
摘要:
以语言目标为条件的紧凑世界模型有望使用一组稀疏的显式“参考锚点”来建立诸如“将红色方块放在蓝色方块左边”这样的关系。我们研究了这些参考何时真正建立了一种关系,并发现了一个陷阱:一个以目标为条件的预测器达到了惊人的0.90的关系读出准确率,但这是“指令转录”,而不是感知。 withholding目标会使其降至随机水平(0.90→0.27,三颗种子),而反事实指令会使预测的锚点在94.5%的时间内遵循“错误”指令(真实场景为2.3%;N = 256)。在三种设置和任务内消融测试中,我们的核心主张描述了混淆因素:当评分数量可从指令转录时(当指令命名答案时)且基本上与非指令输入的预测性无关时,就会发生“指令泄漏”。我们的桌面和外部BabyAI基准会泄漏,而一个指令命名“参考物”的语言表前向动力学世界模型则不会,直到指令被扩充以命名方向;并且降低动作永远不会增加泄漏,这与预测器竞争的预测相反。诊断规定了修复方法:将目标排除在动力学之外(它属于规划器的成本)并监督“读取”路径,恢复真正的、与指令无关的基础(0.88,有目标和无目标时相同)。检测协议和补救措施适用于任何指令命名评分数量的以目标为条件的世界模型。
英文摘要:
Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor reaches a striking $0.90$ relation-readout accuracy, yet this is \emph{instruction transcription}, not perception. Withholding the goal collapses it to chance ($0.90\!\to\!0.27$, three seeds) and a counterfactual instruction makes the predicted anchors follow the \emph{false} instruction $94.5\%$ of the time (true scene $2.3\%$; $N{=}256$). Tested across three settings and a within-task ablation, our central claim characterizes the confound: \textbf{instruction leakage occurs when the scored quantity is transcribable from the instruction (when the instruction names the answer) and is essentially independent of how predictive the non-instruction inputs are.} Our tabletop and the external BabyAI benchmark leak, whereas a Language-Table forward-dynamics world model whose instruction names \emph{referents} does not, until the instruction is augmented to name the direction; and degrading the action never increases leakage, the opposite of what predictor-competition predicts. The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner's cost) and supervise the \emph{read} path, recovering genuine, instruction-independent grounding ($0.88$, identical with and without the goal). The detection protocol and remedy apply to any goal-conditioned world model whose instruction names the scored quantity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。