自我调节与请求干预
Self-Regulation and Requesting Interventions
AI总结：
本文提出一种离线框架，结合基于LLM的过程奖励模型与表格强化学习训练辅助策略，使智能体在有限干预预算下学会何时请求帮助，从而在避免过度自信和过度求助之间取得最优平衡。
摘要：
人类智能涉及元认知能力，如自我调节、识别自身局限以及仅在必要时寻求帮助。尽管LLM智能体在许多领域表现出色，但它们往往缺乏这种意识。过度自信的智能体可能面临灾难性失败，而过度寻求帮助的智能体则阻碍效率。一个关键挑战是使干预预算为$C$的智能体能够决定何时请求帮助。在本文中，我们提出一个离线框架，通过将基于LLM的过程奖励模型（PRMs）与表格强化学习相结合，训练一个“辅助”策略来请求干预，例如调用更强大的模型或测试时计算。利用离线收集的状态转移，我们用PRMs对最优干预时机进行评分，并在这些带标签的轨迹上训练辅助模型。这种离线方法显著减少了训练期间昂贵的干预调用。此外，PRMs与表格强化学习的集成增强了对离线策略数据的鲁棒性，同时避免了深度强化学习的低效性。我们通过实验发现，我们的方法能够实现最优的辅助行为。
英文摘要：
Human intelligence involves metacognitive abilities like self-regulation, recognizing limitations, and seeking assistance only when needed. While LLM Agents excel in many domains, they often lack this awareness. Overconfident agents risk catastrophic failures, while those that seek help excessively hinder efficiency. A key challenge is enabling agents with a limited intervention budget $C$ is to decide when to request assistance. In this paper, we propose an offline framework that trains a "helper" policy to request interventions, such as more powerful models or test-time compute, by combining LLM-based process reward models (PRMs) with tabular reinforcement learning. Using state transitions collected offline, we score optimal intervention timing with PRMs and train the helper model on these labeled trajectories. This offline approach significantly reduces costly intervention calls during training. Furthermore, the integration of PRMs with tabular RL enhances robustness to off-policy data while avoiding the inefficiencies of deep RL. We empirically find that our method delivers optimal helper behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
