arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.08812cs.LG
收藏

TRAM: 测试时风险适应与代理混合

TRAM: Test-Time Risk Adaptation with Mixture of Agents

UT Austin(得克萨斯大学);University of Central Florida(中央佛罗里达大学);MIT(麻省理工学院);UMD(大学公园分校)

Mohamad Chehade, Amrit Singh Bedi, Souradip Chakraborty, Amy Zhang, Hao Zhu

AI总结:

本文研究了在部署时无需更新的零更新适应问题,提出TRAM方法通过混合代理评估源策略的风险调整分数,以降低部署风险并保持奖励。

摘要:

部署的强化学习代理常面临在训练后才指定的安全要求,如新的危险地图、修订的风险阈值或行为对齐约束。我们研究零更新部署时适应,其中固定的风险中性源策略库在新的奖励-风险权衡下被重用。我们提出TRAM(通过代理混合的测试时风险适应),一种源评分的组合规则,该规则在目标奖励和基于占用的部署风险下评估每个源策略,然后使用风险调整的源评分选择动作。不同于训练时与固定替代物(如回报方差)绑定的风险敏感方法,TRAM支持在测试时指定的空间屏障暴露、与参考行为的偏离以及局部波动风险。我们明确将TRAM作为替代方法:它不解决拼接策略的完整占用控制问题,但允许一个可测量的源壳匹配项,将源评分风险与实际风险联系起来。在网格世界、MuJoCo Reacher、Safety-Gymnasium和LLM对齐设置中的实验表明,TRAM在不需测试时任何参数更新的情况下减少了部署风险,同时保持了奖励。

英文摘要:

Deployed reinforcement learning agents often face safety requirements that are specified only after training: new hazard maps, revised risk thresholds, or behavioral alignment constraints. We study zero-update deployment-time adaptation in which a fixed library of risk-neutral source policies must be reused under a newly specified reward-risk tradeoff. We propose TRAM (Test-time Risk Adaptation with Mixture of Agents), a source-scored composition rule that evaluates each source under the target reward and an occupancy-based deployment risk, then selects actions using risk-adjusted source scores. Unlike training-time risk-sensitive methods tied to a fixed surrogate such as return variance, TRAM supports spatial barrier exposure, divergence to a reference behavior, and local volatility risks specified at test time. We make the surrogate nature of the method explicit: TRAM is not claimed to solve the full occupancy-control problem of the stitched policy, but it admits a measurable source-hull mismatch term connecting source-scored risk to realized risk. Experiments in gridworlds, MuJoCo Reacher, Safety-Gymnasium, and an inference-time LLM alignment setting show that TRAM improves deployment risk while preserving reward and requires no parameter updates at test time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑