arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.04466cs.CLcs.AI
收藏

语言模型中的即时任务表示与分布式任务表示

Just-in-time and distributed task representations in language models

Google DeepMind(谷歌DeepMind)

Yuxuan Li, Declan Campbell, Stephanie C. Y. Chan, Andrew Kyle Lampinen

AI总结:

研究揭示了语言模型通过即时任务表示和持续敏感性来实时处理新任务的机制。

摘要:

许多语言模型令人印象深刻的性能来源于其上下文学习能力:基于指令或示例,它们可以在不更新权重的情况下推断并执行新任务。在本工作中,我们研究了语言模型中新任务的表示何时形成,以及这些表示在上下文过程中如何变化。我们研究了两种不同的任务表示:一种是“可转移”的表示——向量表示可以将任务上下文转移到另一个模型实例,即使没有完整的提示;另一种是更简单的高层次任务类别表示。我们发现,可转移的任务表示在非单调和偶发的方式下演变,而任务身份表示在整个上下文中保持不变。具体来说,可转移的任务表示表现出双重局部性。当上下文中提供更多示例时,它们能够成功地浓缩证据。但这一证据累积过程在序列维度上表现出强烈的时序局部性,只在特定的标记上出现——尽管任务身份在整个上下文中都能可靠地解码。在某些情况下,可转移的任务表示也表现出语义局部性,捕捉到一个小的任务“范围”,例如一个独立的子任务。因此,语言模型通过一种惰性、持续的任务敏感性和一种主动、即时的任务表示来实时表示新任务。

英文摘要:

Many of language models' impressive capabilities originate from their in-context learning: based on instructions or examples, they can infer and perform new tasks without weight updates. In this work, we investigate when representations for new tasks are formed in language models, and how these representations change over the course of context. We study two different task representations: those that are ''transferrable'' -- vector representations that can transfer task contexts to another model instance, even without the full prompt -- and simpler representations of high-level task categories. We show that transferrable task representations evolve in non-monotonic and sporadic ways, while task identity representations persist throughout the context. Specifically, transferrable task representations exhibit a two-fold locality. They successfully condense evidence when more examples are provided in the context. But this evidence accrual process exhibits strong temporal locality along the sequence dimension, coming online only at certain tokens -- despite task identity being reliably decodable throughout the context. In some cases, transferrable task representations also show semantic locality, capturing a small task ''scope'' such as an independent subtask. Language models thus represent new tasks on the fly through both an inert, sustained sensitivity to the task and an active, just-in-time representation to support inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑