arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.19346cs.LGcs.CL
收藏

多项式回归作为通过微调和对齐理解上下文学习的任务

Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment

University of California, Berkeley(加州大学伯克利分校)

Max Wilcoxson, Morten Svendgård, Ria Doshi, Dylan Davis, Reya Vir, Anant Sahai

AI总结:

本文提出单变量多项式回归作为函数类,以弥补线性回归等简单函数类在结构上的不足,从而在具备上下文学习能力的模型中研究提示与对齐问题,并实现清晰的可视化与理解。

摘要:

简单函数类已作为玩具问题出现,以更好地理解大型语言模型中基于transformer架构的上下文学习。但先前提出的简单函数类(如线性回归或多层感知机)缺乏所需的结构,无法在具备上下文学习能力的模型中探索提示和对齐等问题。我们提出将单变量多项式回归作为一个函数类,它恰好足够丰富以研究提示和对齐,同时允许我们清晰地可视化和理解其内部机制。

英文摘要:

Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑