arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.12580cs.LGcs.AIstat.ML
收藏

指针值检索:理解神经网络泛化极限的新基准

Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization

Google Research(谷歌研究院);Cornell University(康奈尔大学);Apple(苹果公司)

Chiyuan Zhang, Maithra Raghu, Jon Kleinberg, Samy Bengio

摘要:

人工神经网络成功的关键是它们的泛化能力。但是,神经网络的泛化是否主要依赖于看到高度相似的训练样本(记忆)?或者,神经网络是否能够进行人类智能风格的推理,如果是的话,在多大程度上?这些仍然是人工神经网络的基本开放问题。在本文中,作为回答这些问题的步骤,我们引入了一个新的基准,指针值检索(PVR)研究神经网络推理的限制。PVR任务套件基于间接推理,间接推理是人类智能的标志,其中第一阶段(任务)包含用于解决第二阶段(任务)的指令。在PVR中,这是通过让任务输入的一部分充当指针来完成的,在不同的输入位置上给出指令,这形成了输出。我们展示了这个简单的规则可以应用于创建一组不同的任务,在不同的输入方式和配置。重要的是,这种间接的使用能够通过分布偏移和增加功能复杂性来系统地改变任务难度。我们对不同的PVR任务进行了详细的实证研究,发现不同数据集大小,神经网络架构和任务复杂性的性能差异很大。此外,通过将分布偏移和增加的功能复杂性,我们开发了细致入微的推理测试,揭示了微妙的失败和令人惊讶的成功,提出了许多有前途的探索方向在这个基准。

英文摘要:

Central to the success of artificial neural networks is their ability to generalize. But does neural network generalization primarily rely on seeing highly similar training examples (memorization)? Or are neural networks capable of human-intelligence styled reasoning, and if so, to what extent? These remain fundamental open questions on artificial neural networks. In this paper, as steps towards answering these questions, we introduce a new benchmark, Pointer Value Retrieval (PVR) to study the limits of neural network reasoning. The PVR suite of tasks is based on reasoning about indirection, a hallmark of human intelligence, where a first stage (task) contains instructions for solving a second stage (task). In PVR, this is done by having one part of the task input act as a pointer, giving instructions on a different input location, which forms the output. We show this simple rule can be applied to create a diverse set of tasks across different input modalities and configurations. Importantly, this use of indirection enables systematically varying task difficulty through distribution shifts and increasing functional complexity. We conduct a detailed empirical study of different PVR tasks, discovering large variations in performance across dataset sizes, neural network architectures and task complexity. Further, by incorporating distribution shift and increased functional complexity, we develop nuanced tests for reasoning, revealing subtle failures and surprising successes, suggesting many promising directions of exploration on this benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑