arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.20264cs.LG
收藏

乐观任务推断用于行为基础模型

Optimistic Task Inference for Behavior Foundation Models

ETH Zürich(苏黎世联邦理工学院);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所)

Thomas Rupf, Marco Bagatella, Marin Vlastelica, Andreas Krause

AI总结:

OpTI-BFM通过测试时与环境的交互,优化奖励函数推断,减少数据和计算开销。

摘要:

行为基础模型(BFMs)能够为任何在测试时直接指定的奖励函数检索高性能策略,通常称为零样本强化学习(RL)。尽管在计算上非常高效,但在数据方面可能不够:作为标准假设,BFMs需要在非可忽略的推断数据集上计算奖励,假设要么有奖励函数的函数形式,要么有显著的标注努力。为了缓解这些限制,我们通过在测试时与环境的交互来解决任务推断问题。我们提出了OpTI-BFM,一种乐观决策准则,直接建模对奖励函数的不确定性,并指导BFMs在任务推断中进行数据收集。形式上,我们通过与线性轮盘算法的直接联系,为训练良好的BFMs提供了一个遗憾界。实证上,我们在已建立的零样本基准上评估OpTI-BFM,并观察到它使继任特征基于的BFMs能够在少量episodes中识别并优化一个未见过的奖励函数,同时具有最小的计算开销。代码可在https://github.com/ThomasRupf/opti-bfm上获得。

英文摘要:

Behavior Foundation Models (BFMs) are capable of retrieving high-performing policy for any reward function specified directly at test-time, commonly referred to as zero-shot reinforcement learning (RL). While this is a very efficient process in terms of compute, it can be less so in terms of data: as a standard assumption, BFMs require computing rewards over a non-negligible inference dataset, assuming either access to a functional form of rewards, or significant labeling efforts. To alleviate these limitations, we tackle the problem of task inference purely through interaction with the environment at test-time. We propose OpTI-BFM, an optimistic decision criterion that directly models uncertainty over reward functions and guides BFMs in data collection for task inference. Formally, we provide a regret bound for well-trained BFMs through a direct connection to upper-confidence algorithms for linear bandits. Empirically, we evaluate OpTI-BFM on established zero-shot benchmarks, and observe that it enables successor-features-based BFMs to identify and optimize an unseen reward function in a handful of episodes with minimal compute overhead. Code is available at https://github.com/ThomasRupf/opti-bfm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑