基于检索的模型的泛化性质
Generalization Properties of Retrieval-based Models
摘要:
许多现代高性能机器学习模型,如GPT-3,主要依赖于按比例放大模型,例如,Transformer网络。同时,一个平行的工作线旨在通过在推理过程中用其他(标记的)实例增强输入实例来提高模型性能。这种增强的示例包括任务特定提示和由非参数组件从训练数据检索的类似示例。值得注意的是,基于检索的方法在广泛的问题上取得了成功,从标准的自然语言处理和视觉任务到蛋白质折叠,最近的许多努力,包括WebGPT和AlphaFold。尽管越来越多的文献展示了这些模型的前景,但这些模型的理论基础仍然没有得到充分的探索。在本文中,我们提出了一个正式的处理检索为基础的模型来表征其泛化能力。特别是,我们专注于两类检索为基础的分类方法:首先,我们分析了一个本地学习框架,采用了显式的本地经验风险最小化的基础上检索的例子为每个输入实例。有趣的是,我们发现,将底层学习任务分解为局部子任务,使模型能够采用低复杂度的参数组件,以确保良好的整体准确性。我们探索的第二类基于检索的方法使用内核方法学习全局模型,直接将输入实例和检索到的示例映射到预测,而无需显式解决局部学习任务。
英文摘要:
Many modern high-performing machine learning models such as GPT-3 primarily rely on scaling up models, e.g., transformer networks. Simultaneously, a parallel line of work aims to improve the model performance by augmenting an input instance with other (labeled) instances during inference. Examples of such augmentations include task-specific prompts and similar examples retrieved from the training data by a nonparametric component. Remarkably, retrieval-based methods have enjoyed success on a wide range of problems, ranging from standard natural language processing and vision tasks to protein folding, as demonstrated by many recent efforts, including WebGPT and AlphaFold. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. In this paper, we present a formal treatment of retrieval-based models to characterize their generalization ability. In particular, we focus on two classes of retrieval-based classification approaches: First, we analyze a local learning framework that employs an explicit local empirical risk minimization based on retrieved examples for each input instance. Interestingly, we show that breaking down the underlying learning task into local sub-tasks enables the model to employ a low complexity parametric component to ensure good overall accuracy. The second class of retrieval-based approaches we explore learns a global model using kernel methods to directly map an input instance and retrieved examples to a prediction, without explicitly solving a local learning task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。