大型语言模型的数值线性代数

中文摘要 AI 辅助

数值线性代数（NLA）一直在通过为科学和工程应用中遇到的基本问题提供求解工具，在推动科学进步方面发挥着至关重要的作用。几十年来，它不断演变以满足由连续的科学发现浪潮所驱动的需求。例如，在20世纪50年代和60年代，大量努力致力于开发求解由快速发展的空气动力学领域涌现出的特征值问题的方法，这导致了LR和QR算法的发现。后来，注意力转向了在计算空气动力学等应用中常见的稀疏线性系统的求解。今天，我们正经历着又一次重大科学进步的浪潮，而数值线性代数再次处于其发展的核心。这股机器学习（ML）浪潮正被证明在科学和工程领域具有彻底的颠覆性。机器学习中的许多工具，特别是大型语言模型（LLMs），都基于矩阵和张量方法。随着我们接近通用人工智能（AGI），很明显矩阵方法将被要求发挥更重要的作用。对于数值线性从业者来说，当前变化的速度使得适应变得特别具有挑战性。这是一篇综述文章，聚焦于机器学习技术，特别关注大型语言模型。它有两个主要目标。第一个是以数值方法专家可理解的方式阐明大型语言模型背后的核心概念。第二个是审视LLM技术所采用的关键数值线性代数概念，同时强调NLA对该领域的几项重要近期贡献。

英文摘要

Numerical Linear Algebra (NLA) has consistently played a vital role in advancing science by providing tools to solve fundamental problems encountered in scientific and engineering applications. Over the decades, it has continually evolved to meet the demands driven by successive waves of scientific discovery. For instance, during the 1950s and 1960s, substantial efforts were devoted to developing methods for solving eigenvalue problems that emerged from the rapidly growing field of aerodynamics. This led to the discovery of the LR and QR algorithms. Later the attention turned to the solution of sparse linear systems that were common in applications like computational aerodynamics. Today we are experiencing yet another wave of major scientific advancement and NLA is once more at the heart of its development. This Machine Learning (ML) wave is proving to be utterly disruptive in science and engineering. Many tools in ML particularly Large Language Models (LLMs) are grounded in matrix and tensor methods. As we are approaching Artificial General Intelligence (AGI), it is clear that matrix methods will be called to play an even more significant role. For the numerical linear practitioner the speed of the current change makes it particularly challenging to adapt. This is a survey article that centers on machine learning techniques, with a particular focus on large language models. It has two main objectives. The first is to clarify the core concepts behind Large Language Models in a manner accessible to specialists in numerical methods. The second is to examine the key Numerical Linear Algebra concepts employed by LLM techniques, while also highlighting several significant recent contributions of NLA to the field.

发表机构

• Qatar Computing Research Institute, Hamad Bin Khalifa University（卡塔尔计算研究所，哈马德·本·哈利法大学）
• University of Minnesota（明尼苏达大学）

机构由 AI 辅助整理，请以论文原文为准。