arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.08009cs.CL
收藏

提示的几何:揭示语言模型中任务适应的不同机制

The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models

New York University(纽约大学);Flatiron Institute(弗拉铁龙研究所);Genentech(基因泰克公司)

Artem Kirsanov, Chi-Ning Chou, Kyunghyun Cho, SueYeon Chung

AI总结:

针对仅解码器语言模型提示灵活性背后机制不明的问题,本文基于统计物理学框架研究不同提示方法对模型表示几何的影响,揭示其通过不同表示机制实现任务适应,强调输入样本和标签语义的作用并发现任务间的协同与干扰,为大语言模型理论理解及高效提示策略开发奠定基础。

摘要:

仅解码器语言模型能够根据输入提示在各种计算任务之间动态切换。尽管提示有许多成功应用,但对这种灵活性背后的内部机制的理解非常有限。在这项工作中,我们研究不同提示方法如何影响这些模型中表示的几何结构。利用基于统计物理学的框架,我们发现各种提示技术虽然性能相似,但通过不同的表示机制实现任务适应。我们的分析强调了输入分布样本和标签语义在少样本上下文学习中的关键作用。我们还展示了不同任务在表示层面上存在协同和干扰相互作用的证据。我们的工作有助于对大型语言模型的理论理解,并为开发更有效的、感知表示的提示策略奠定基础。

英文摘要:

Decoder-only language models have the ability to dynamically switch between various computational tasks based on input prompts. Despite many successful applications of prompting, there is very limited understanding of the internal mechanism behind such flexibility. In this work, we investigate how different prompting methods affect the geometry of representations in these models. Employing a framework grounded in statistical physics, we reveal that various prompting techniques, while achieving similar performance, operate through distinct representational mechanisms for task adaptation. Our analysis highlights the critical role of input distribution samples and label semantics in few-shot in-context learning. We also demonstrate evidence of synergistic and interfering interactions between different tasks on the representational level. Our work contributes to the theoretical understanding of large language models and lays the groundwork for developing more effective, representation-aware prompting strategies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑