Transformers have shown inconsistent success in AI planning tasks, and theoretical understanding of when generalization should be expected has been limited. We take important steps towards addressing this gap by analyzing the ability of decoder-only models to verify whether a given plan correctly solves a given planning instance. To analyse the general setting where the number of objects -- and thus the effective input alphabet -- grows at test time, we introduce C*-RASP, an extension of C-RASP designed to establish length generalization guarantees for transformers under the simultaneous growth in sequence length and vocabulary size. Our results identify a large class of classical planning domains for which transformers can provably learn to verify long plans, and structural properties that significantly affects the learnability of length generalizable solutions. Empirical experiments corroborate our theory.
Can Transformers Learn to Verify During Backtracking Search?
Yin Jun Phua (corresponding author) Institute of Science Tokyo, 2-12-1 Ookayama, Meguro-ku, Tokyo 152-8550, Japan Tony Ribeiro Centrale Nantes, CNRS, Laboratoire des Sciences du Num\'erique de Nantes, LS2N, UMR 6004, F-44000 Nantes, France National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan Steelous Protocol, 8-20-32, Ginza, Chuo-ku, Tokyo 104-0061, Japan Tuan Nguyen Hanoi University of Science;Technology, No. 1 Dai Co Viet, Hai Ba Trung, Ha Noi, Vietnam Katsumi Inoue National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan
Enterprise Resource Planning Using Multi-type Transformers in Ferro-Titanium Industry
Department of Mathematics and Industrial Engineering(数学与工业工程系);Polytechnique Montreal(蒙特利尔理工学院);Department of Mechanical, Industrial and Aerospace Engineering(机械、工业和航空航天工程系);Concordia University(Concordia大学)
Fast and scalable retrosynthetic planning with a transformer neural network and speculative beam search
SUPSI, IDSIA(SUPSI、IDSIA);Pfizer Research and Development(辉瑞研发);Institute for Digital Technologies for Personalized Healthcare(个性化医疗数字技术研究所);KAUST(王国立科技大学);Center for Generative AI(生成人工智能中心)
Plansformer: Generating Symbolic Plans using Transformers
Artificial Intelligence Institute, University of South Carolina(南卡罗来纳大学人工智能研究所);IBM Research(IBM研究院);University of Parma(帕尔马大学);University of Brescia(布雷西亚大学)
Washington University in St. Louis(圣路易斯华盛顿大学);Duke University(杜克大学);Stevens Institute of Technology(史蒂文斯理工学院);University of Michigan(密歇根大学);Northeastern University(东北大学)