arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.02536cs.LGcs.AI
收藏

Transformers发现了一个利用局部注意力和网格式问题表示的基本计算系统

Transformers discover an elementary calculation system exploiting local attention and grid-like problem representation

University of Padova(帕多瓦大学)

Samuel Cognolato, Alberto Testolin

摘要:

数学推理是人类智力最令人印象深刻的成就之一,但对于人工智能系统来说仍然是一个艰巨的挑战。在这项工作中,我们探索现代深度学习架构是否可以通过发现有效的算术过程来学习解决符号加法任务。虽然这个问题乍一看似乎微不足道,但将算术知识推广到涉及更多项的运算,可能由更长的数字序列组成,这对神经网络来说是极具挑战性的。在这里,我们表明,通用Transformers配备了本地注意力和自适应停机机制,可以学习利用外部,网格状的存储器进行多位数相加。即使在训练分布之外需要外推的问题进行测试时,所提出的模型也达到了显着的准确性;最值得注意的是,它通过发现类似人类的计算策略(如位置价值对齐)来实现。

英文摘要:

Mathematical reasoning is one of the most impressive achievements of human intellect but remains a formidable challenge for artificial intelligence systems. In this work we explore whether modern deep learning architectures can learn to solve a symbolic addition task by discovering effective arithmetic procedures. Although the problem might seem trivial at first glance, generalizing arithmetic knowledge to operations involving a higher number of terms, possibly composed by longer sequences of digits, has proven extremely challenging for neural networks. Here we show that universal transformers equipped with local attention and adaptive halting mechanisms can learn to exploit an external, grid-like memory to carry out multi-digit addition. The proposed model achieves remarkable accuracy even when tested with problems requiring extrapolation outside the training distribution; most notably, it does so by discovering human-like calculation strategies such as place value alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑