arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.08164stat.MLcs.AIcs.LG
收藏

论Transformer架构的局限性

On Limitations of the Transformer Architecture

Google(谷歌公司)

Binghui Peng, Srini Narayanan, Christos Papadimitriou

AI总结:

本文利用通信复杂度理论证明Transformer层在函数定义域较大时无法组合函数,并通过实例和计算复杂度猜想说明其在组合任务上的根本局限,揭示了大语言模型幻觉的可能根源。

摘要:

大语言模型(LLMs)中幻觉产生的根本原因是什么?我们利用通信复杂度(Communication Complexity)证明,当函数的定义域足够大时,Transformer层无法组合函数(例如,在一个家谱中识别一个人的祖父母);我们通过示例表明,即使定义域相当小,这种能力缺失在经验上已经显现。我们还指出,若实例规模足够大,并假设计算复杂度领域中某些被广泛接受的猜想成立,那么若干作为所谓组合任务核心的数学任务——这些任务被认为对LLMs而言是困难的——不太可能由Transformer解决。

英文摘要:

What are the root causes of hallucinations in large language models (LLMs)? We use Communication Complexity to prove that the Transformer layer is incapable of composing functions (e.g., identify a grandparent of a person in a genealogy) if the domains of the functions are large enough; we show through examples that this inability is already empirically present when the domains are quite small. We also point out that several mathematical tasks that are at the core of the so-called compositional tasks thought to be hard for LLMs are unlikely to be solvable by Transformers, for large enough instances and assuming that certain well accepted conjectures in the field of Computational Complexity are true.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑