上下文强化学习的任务专业化微调
Task Specialization Fine-Tuning for Contextual Reinforcement Learning
- Nanyang Technological University（南洋理工大学）
- MIT（麻省理工学院）
- UIUC（伊利诺伊大学厄巴纳-香槟分校）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对上下文强化学习的任务专业化微调难题，提出TSFT框架，通过整数线性规划分配微调预算，在多领域实验中提升了任务覆盖性能。
AI中文摘要：
上下文强化学习（CRL）旨在通过最大化相关任务上下文空间中的任务覆盖范围，对经典强化学习（RL）进行泛化。现有研究通常从头开始训练，要么针对单一策略采用多任务学习，要么策略性训练多个策略；我们则倡导一种统一的替代方案：预训练一个初始性能良好的单一策略，随后针对任务专业化微调多个策略。然而，这一新范式带来了独特挑战，如异质边际回报和样本效率低下。由此产生一个关键研究问题：给定一个预训练策略和有限预算，每个任务区域应接受多少微调才能实现样本高效的CRL？为此，我们提出任务专业化微调（TSFT），这是一个在线框架，通过简单的参数模型预测微调性能，并通过整数线性规划精确求解由此产生的离散预算分配问题。在组合优化、连续控制和LLM微调等不同决策领域开展的大量实验表明，TSFT在任务覆盖范围上显著优于基线方法，并接近理想性能。我们的工作为基于模型的CRL开辟了新方向，契合现代预训练-微调时代的发展趋势。
英文摘要：
Contextual Reinforcement Learning (CRL) seeks to generalize classical RL by maximizing task coverage across a context space of related tasks. While prior works often train from scratch and rely on either multi-task learning for a single policy or strategically training multiple policies, we advocate for a unified alternative: pretraining a single policy with good initial performance, followed by fine-tuning multiple policies for task specialization. This new paradigm, however, introduces unique challenges, such as heterogeneous marginal returns and sample inefficiency. This raises a critical research question: given a pretrained policy and a constrained budget, how much fine-tuning should each task region receive to enable sample-efficient CRL? To this end, we propose Task Specialization Fine-Tuning (TSFT), an online framework that predicts fine-tuning performance with a simple parametric model and exactly solves the resulting discrete budget allocation problem via integer linear programming. Extensive experiments across diverse decision domains, including combinatorial optimization, continuous control, and LLM fine-tuning, demonstrate that TSFT significantly outperforms baselines in task coverage and approaches oracle performance. Our work charts a new direction for model-based CRL, aligning with the modern pretrain-finetune era.
