利用序列模型学习高精度最小二乘算法
Towards Learning High-Precision Least Squares Algorithms with Sequence Models
AI总结:
针对最小二乘问题,本文提出多项式序列架构与高精度训练方案,使模型可学习近机器精度的梯度下降迭代,并显著降低 MSE 与分布外泛化差距。
摘要:
本文研究序列模型能否学会在最小二乘这一基础问题上执行数值算法,例如梯度下降。我们的目标是继承数值分析中标准算法的两个性质:(1)机器精度,即希望获得精度接近浮点误差的解;(2)数值通用性,即希望它们能够广泛适用于不同问题实例。我们发现,以往使用 Transformers 的方法无法满足这些标准,并指出现有架构和训练流程中存在的局限。首先,我们证明 softmax Transformers 难以执行高精度乘法,这使其无法精确学习数值算法。其次,我们识别出另一类完全由多项式构成的架构,它们能够高效表示高精度梯度下降迭代。最后,我们研究训练过程中的精度瓶颈,并通过一种降低随机梯度噪声的高精度训练方案加以解决。该方案使我们能够训练两种多项式架构——门控卷积和线性注意力——以在最小二乘问题上执行梯度下降迭代。我们首次展示了训练至接近机器精度的能力。在迭代应用时,我们的模型获得的 MSE 比端到端训练的标准 Transformers 低 100,000 倍,并且在分布外问题上的泛化差距小 10,000 倍。我们在面向最小二乘的数值算法端到端学习方面取得了进展。
英文摘要:
This paper investigates whether sequence models can learn to perform numerical algorithms, e.g. gradient descent, on the fundamental problem of least squares. Our goal is to inherit two properties of standard algorithms from numerical analysis: (1) machine precision, i.e. we want to obtain solutions that are accurate to near floating point error, and (2) numerical generality, i.e. we want them to apply broadly across problem instances. We find that prior approaches using Transformers fail to meet these criteria, and identify limitations present in existing architectures and training procedures. First, we show that softmax Transformers struggle to perform high-precision multiplications, which prevents them from precisely learning numerical algorithms. Second, we identify an alternate class of architectures, comprised entirely of polynomials, that can efficiently represent high-precision gradient descent iterates. Finally, we investigate precision bottlenecks during training and address them via a high-precision training recipe that reduces stochastic gradient noise. Our recipe enables us to train two polynomial architectures, gated convolutions and linear attention, to perform gradient descent iterates on least squares problems. For the first time, we demonstrate the ability to train to near machine precision. Applied iteratively, our models obtain 100,000x lower MSE than standard Transformers trained end-to-end and they incur a 10,000x smaller generalization gap on out-of-distribution problems. We make progress towards end-to-end learning of numerical algorithms for least squares.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。