基于摊销低秩适应的模型强化学习
Amortized Low-Rank Adaptation for Model-Based Reinforcement Learning
- University of Texas at Austin（德克萨斯大学奥斯汀分校）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对世界模型在测试环境中的适应问题，提出CLAW方法，利用超网络生成低秩适配器，在少量数据下实现高效适应，优于梯度适应和上下文学习。
AI中文摘要：
世界模型让智能体能够通过预测其行为的后果来进行规划，但环境的变化可能使这些模型变得不准确。我们研究的问题是在已知环境族中，仅使用少量交互回合，将世界模型适应到未知的测试时环境。现有方法在计算成本与表达力（即方法能产生的模型范围）之间进行权衡。例如，上下文学习计算成本低但表达力有限，而基于梯度的适应表达力强但计算成本高。我们提出了CLAW（世界模型的上下文条件低秩适应），通过使用超网络在测试时生成低秩（LoRA）适配器来解决这一权衡。在预训练期间，我们模拟对多种环境的适应，并联合训练超网络和基础世界模型。在测试时，我们冻结基础模型，并使用超网络的前向传播从一小批测试时转换中生成适配器。我们在动力学、具身和奖励各异的运动与操作环境族中评估CLAW。我们表明，仅使用数秒的测试时数据，CLAW在在线适应中优于基于梯度的适应和上下文学习。我们还表明，CLAW在数据稀缺情况下避免了过拟合，其优势来自表达力强的适配器而非上下文条件，并且将超网络与基础模型联合预训练优于事后训练。
英文摘要：
World models let agents plan by predicting the consequences of their actions, but changes in the environment can make them inaccurate. We study the problem of adapting a world model to an unknown test-time environment, drawn from a known environment family, using only a few episodes of interaction. Existing approaches trade off computational cost against expressivity, i.e., the range of models a method can produce. For example, in-context learning is computationally cheap but limited in expressivity, and gradient-based adaptation is expressive but computationally expensive. We present CLAW (Context-conditioned Low-rank Adaptation of World models), which addresses this tradeoff by using a hypernetwork to generate low-rank (LoRA) adapters at test time. During pretraining, we simulate adaptation to a variety of environments and jointly train the hypernetwork and base world model. At test time, we freeze the base model and use a forward pass of the hypernetwork to generate adapters from a small batch of test-time transitions. We evaluate CLAW in locomotion and manipulation environment families that vary in dynamics, embodiment, and reward. We show that, using only seconds of test-time data, CLAW outperforms gradient-based adaptation and in-context learning during online adaptation. We also show that CLAW avoids overfitting in data-scarce regimes, that its advantage comes from the expressive adapters rather than context conditioning, and that pretraining the hypernetwork jointly with the base model outperforms training it post hoc.
