arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.07772cs.CLcs.LG
收藏

重新审视上下文学习与梯度下降

In-context Learning and Gradient Descent Revisited

The Blavatnik School of Computer Science(布拉瓦尼克计算机科学学院);Tel Aviv University(特拉维夫大学)

Gilad Deutch, Nadav Magar, Tomer Bar Natan, Guy Dar

AI总结:

本文重新检验了上下文学习与梯度下降的对应关系,发现现有评估存在缺陷,并提出尊重层因果关系的梯度下降优化方法以显著提升相似度。

摘要:

上下文学习(ICL)在少样本学习任务中表现出令人印象深刻的结果,但其底层机制仍未完全被理解。最近一系列研究表明,ICL 隐式地执行基于梯度下降(GD)的优化。尽管这一观点很有吸引力,但大部分研究集中在简化设置上,即仅优化浅层模型的参数。在本工作中,我们重新审视了在现实自然语言处理任务和模型上 ICL 与 GD 对应关系的证据。我们发现评估中存在不足,既包括有问题的度量指标,也包括不充分的基线。我们表明,令人惊讶的是,即使未训练的模型也能获得与 ICL-GD 相似度相当的分数,尽管它们并未表现出 ICL。接下来,我们探索了 ICL 与 GD 之间在信息流经模型过程上的一个重大差异,我们将其称为层因果关系。我们提出了一种简单的基于 GD 的优化过程,该过程尊重层因果关系,并表明它能显著提高相似度分数。

英文摘要:

In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism is still not fully understood. A recent line of work suggests that ICL performs gradient descent (GD)-based optimization implicitly. While appealing, much of the research focuses on simplified settings, where the parameters of a shallow model are optimized. In this work, we revisit evidence for ICL-GD correspondence on realistic NLP tasks and models. We find gaps in evaluation, both in terms of problematic metrics and insufficient baselines. We show that surprisingly, even untrained models achieve comparable ICL-GD similarity scores despite not exhibiting ICL. Next, we explore a major discrepancy in the flow of information throughout the model between ICL and GD, which we term Layer Causality. We propose a simple GD-based optimization procedure that respects layer causality, and show it improves similarity scores significantly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑