arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.14219cs.LGcs.AI
收藏

理解用于序贯决策的预训练 Transformer 的训练与泛化

Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Making

Imperial College London(伦敦帝国学院);HKUST(GZ)(香港科技大学(广州));University of North Carolina(北卡罗来纳大学)

Hanzhao Wang, Yu Pan, Fupeng Sun, Shang Liu, Kalyan Talluri, Guanting Chen, Xiaocheng Li

AI总结:

本文针对无转移概率的序贯决策问题,提出将 Transformer 生成序列纳入训练以解决分布外问题,并揭示其探索机制及相对 UCB、Thompson sampling 的优势。

摘要:

本文研究用于一类序贯决策问题的监督式预训练 Transformer。所考虑的问题类别是强化学习一般表述的一个子集,因为其中不存在转移概率矩阵;尽管看似具有限制性,但该子类问题涵盖了老虎机、动态定价和报童问题等特殊情形。这种结构使得在预训练阶段能够使用最优行动/决策,而这种使用也为预训练 Transformer 的训练和泛化提供了新的见解。我们首先指出,Transformer 模型的训练可被视为一个表演型预测问题,而现有方法和理论在很大程度上忽视或无法解决分布外问题。我们提出了一种自然的解决方案,在训练过程中纳入由 Transformer 生成的行动序列,该方案在数值和理论上都具有更好的性质。在所考虑任务中能够获得最优行动,这也使我们能够将预训练 Transformer 作为一种算法来分析其性质,并解释它为何可能缺乏探索,以及这一问题如何能够被自动解决。在数值实验方面,我们将预训练 Transformer 相对于 UCB 和 Thompson sampling 等结构化算法的优势归纳为三种情形:(i)它能更好地利用预训练数据中的先验知识;(ii)它能够巧妙处理结构化算法所遭受的模型误设问题;(iii)对于 T≤50 等较短时间跨度,它表现得更为贪婪,并比那些为渐近最优性而设计的结构化算法获得低得多的遗憾值。

英文摘要:

In this paper, we consider the supervised pre-trained transformer for a class of sequential decision-making problems. The class of considered problems is a subset of the general formulation of reinforcement learning in that there is no transition probability matrix; though seemingly restrictive, the subset class of problems covers bandits, dynamic pricing, and newsvendor problems as special cases. Such a structure enables the use of optimal actions/decisions in the pre-training phase, and the usage also provides new insights for the training and generalization of the pre-trained transformer. We first note the training of the transformer model can be viewed as a performative prediction problem, and the existing methods and theories largely ignore or cannot resolve an out-of-distribution issue. We propose a natural solution that includes the transformer-generated action sequences in the training procedure, and it enjoys better properties both numerically and theoretically. The availability of the optimal actions in the considered tasks also allows us to analyze the properties of the pre-trained transformer as an algorithm and explains why it may lack exploration and how this can be automatically resolved. Numerically, we categorize the advantages of pre-trained transformers over the structured algorithms such as UCB and Thompson sampling into three cases: (i) it better utilizes the prior knowledge in the pre-training data; (ii) it can elegantly handle the misspecification issue suffered by the structured algorithms; (iii) for short time horizon such as $T\le50$, it behaves more greedy and enjoys much better regret than the structured algorithms designed for asymptotic optimality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑