arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.01003cs.LGcs.CL
收藏

竞争动力学塑造了上下文学习的算法相变

Competition Dynamics Shape Algorithmic Phases of In-Context Learning

Harvard University(哈佛大学);NTT Research, Inc.(NTT研究公司);University of Michigan, Ann Arbor(密歇根大学安阿伯分校)

Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, Hidenori Tanaka

AI总结:

本文提出了一种学习模拟有限混合Markov链的合成序列建模任务以统一研究ICL,发现模型行为可分解为四种竞争算法,实验条件变化会导致算法间的相变,表明ICL是多种算法的混合而非单一能力。

摘要:

上下文学习(ICL)显著扩展了大型语言模型的通用性,使其能够仅利用输入的上下文来适应新任务。这促使了一系列论文分析易于处理的合成领域,并假设了可能构成 ICL 基础的精确机制。然而,这些研究使用了相对不同的设置,且通常缺乏序列建模特性,使得这些研究得出的见解具有多大普适性尚不清楚。受此启发,我们提出了一项合成序列建模任务,该任务涉及学习模拟有限混合的 Markov 链。正如我们所展示的,在该任务上训练的模型重现了关于 ICL 的大多数已知结果,从而为研究该概念提供了一个统一的环境。基于这一设置,我们证明可以通过将模型行为分解为四种广义算法来解释它,这些算法将模糊检索与推断方法同上下文的 unigram 或 bigram 统计信息结合起来。这些算法参与竞争动力学以主导模型行为,而精确的实验条件决定了哪种算法最终取代其他算法:例如,我们发现仅仅改变上下文大小或训练量就会在不同算法主导模型行为之间产生(有时是急剧的)转变,揭示了一种解释 ICL 瞬态特性的机制。从这个意义上说,我们认为 ICL 最好被视为不同算法的混合,每种算法都有其自身的特点,而不是一种单一的能力。这也意味着做出在所有设置中普遍适用的关于 ICL 的一般性主张可能是不可行的。

英文摘要:

In-Context Learning (ICL) has significantly expanded the general-purpose nature of large language models, allowing them to adapt to novel tasks using merely the inputted context. This has motivated a series of papers that analyze tractable synthetic domains and postulate precise mechanisms that may underlie ICL. However, the use of relatively distinct setups that often lack a sequence modeling nature to them makes it unclear how general the reported insights from such studies are. Motivated by this, we propose a synthetic sequence modeling task that involves learning to simulate a finite mixture of Markov chains. As we show, models trained on this task reproduce most well-known results on ICL, hence offering a unified setting for studying the concept. Building on this setup, we demonstrate we can explain a model's behavior by decomposing it into four broad algorithms that combine a fuzzy retrieval vs. inference approach with either unigram or bigram statistics of the context. These algorithms engage in a competition dynamics to dominate model behavior, with the precise experimental conditions dictating which algorithm ends up superseding others: e.g., we find merely varying context size or amount of training yields (at times sharp) transitions between which algorithm dictates the model behavior, revealing a mechanism that explains the transient nature of ICL. In this sense, we argue ICL is best thought of as a mixture of different algorithms, each with its own peculiarities, instead of a monolithic capability. This also implies that making general claims about ICL that hold universally across all settings may be infeasible.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑