arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.01685cs.CLcs.AI
收藏

语言模型如何组合函数?

How Do Language Models Compose Functions?

Department of Computer Science(计算机科学系)

Apoorv Khandelwal, Ellie Pavlick

AI总结:

研究语言模型在解决两跳事实回忆任务时的组合机制,发现现代LLM存在组合性差距,通过分析残差流发现两种处理机制,并发现嵌入空间几何与所用机制密切相关。

摘要:

尽管大型语言模型(LLM)似乎在解决组合性任务方面能力日益增强,但它们是否通过组合性机制来完成仍是一个开放性问题。在本工作中,我们研究了前馈LLM如何解决两跳事实回忆任务,该任务可以表示为$g(f(x))$的组合性形式。我们首先确认现代LLM仍然存在

英文摘要:

While large language models (LLMs) appear to be increasingly capable of solving compositional tasks, it is an open question whether they do so using compositional mechanisms. In this work, we investigate how feedforward LLMs solve two-hop factual recall tasks, which can be expressed compositionally as $g(f(x))$. We first confirm that modern LLMs continue to suffer from the "compositionality gap", i.e. their ability to compute both $z = f(x)$ and $y = g(z)$ does not entail their ability to compute the composition $y = g(f(x))$. We then decode residual stream representations and identify two processing mechanisms: one which solves tasks $\textit{compositionally}$, computing $f(x)$ along the way to $g(f(x))$, and one which solves them $\textit{directly}$, without any detectable signature of the intermediate variable $f(x)$. Finally, we find that embedding space geometry is strongly related to which mechanism is employed, where the idiomatic mechanism is dominant when tasks are represented by translations from $x$ to $g(f(x))$ in the embedding spaces. We fully release our data and code at: https://github.com/apoorvkh/composing-functions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑