黑暗中的询问:部分可观测性下的不确定性门控语言模型辅助
ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability
AI总结:
研究部分可观测性下强化学习智能体,指出普通不确定性门控方法失败原因是上下文问题。提出ASK+,为语言模型提供轨迹感知上下文和结构化推理,证明预测熵信号可行,在多环境中取得更好效果,强调提示设计重要性。
摘要:
在部分可观测性下运行的强化学习智能体必须基于不完整信息行动,使其成为从小语言模型(SLM)获得指导的自然候选者。然而,将SLM指导集成到这种设置中已被证明很困难:在所有测试环境中,普通的不确定性门控方法实现的覆盖率为零或接近零,这意味着SLM几乎从不贡献独立行动。我们将这种失败归因于纯粹的自我中心提示,它为真正的推理提供的上下文不足,并将其识别为上下文问题而非能力问题。我们提出了ASK+,它为SLM提供轨迹感知上下文(部分揭示的地图、访问过的位置和行动历史)和结构化的思维链推理,将其从被动的冗余检查转变为一个更具信息性的顾问,偶尔会纠正策略。我们进一步证明,用于选择性查询的预测熵信号测量的是行动不确定性而非状态不确定性,并且在部分可观测马尔可夫决策过程(POMDP)中仍然具有信息性,使得不确定性门控辅助在完全可观测设置之外也可行。有状态提示带来了显著的收益:在DoorKey上,普通的ASK与近端策略优化算法(PPO)匹配(均为89%),ASK+的成功率达到93%;在FourRooms上,成功率从53%攀升至70%;在HigherLower上,准确率达到73.7%,与仅使用SLM的上限匹配。在所有环境中,Qwen3.5 - 2B匹配或超过Qwen3.5 - 4B,证实了提示设计和选择性门控主导了模型规模的影响,无需大型模型就能实现指导。
英文摘要:
Reinforcement learning agents operating under partial observability must act on incomplete information, making them natural candidates for guidance from small language models (SLMs) that carry broad reasoning priors. Yet integrating SLM guidance into this setting has proven difficult: across all test environments, vanilla uncertainty-gated approaches achieve an overwrite rate at or near zero, meaning the SLM almost never contributes an independent action. We trace this failure to the bare egocentric prompt, which provides insufficient context for genuine reasoning, and identify it as a context problem rather than a capacity problem. We propose ASK+, which supplies the SLM with trajectory-aware context (a partially revealed map, visited positions, and action history) and structured chain-of-thought reasoning, converting it from a passive redundancy check into a more informative consultant that occasionally corrects the policy. We further establish that the predictive entropy signal used for selective querying measures action uncertainty rather than state uncertainty and remains informative in POMDPs, making uncertainty-gated assistance viable beyond fully observable settings. The stateful prompt drives substantial gains: on DoorKey, where vanilla ASK matches PPO (both 89%), ASK+ reaches 93% success; on FourRooms, success climbs from 53% to 70%; on HigherLower, accuracy reaches 73.7%, matching the SLM-only upper bound. Across all environments, Qwen3.5-2B matches or exceeds Qwen3.5-4B, confirming that prompt design and selective gating dominate the impact of model scale, enabling guidance without large models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。