Central to the success of artificial neural networks is their ability to generalize. But does neural network generalization primarily rely on seeing highly similar training examples (memorization)? Or are neural networks capable of human-intelligence styled reasoning, and if so, to what extent? These remain fundamental open questions on artificial neural networks. In this paper, as steps towards answering these questions, we introduce a new benchmark, Pointer Value Retrieval (PVR) to study the limits of neural network reasoning. The PVR suite of tasks is based on reasoning about indirection, a hallmark of human intelligence, where a first stage (task) contains instructions for solving a second stage (task). In PVR, this is done by having one part of the task input act as a pointer, giving instructions on a different input location, which forms the output. We show this simple rule can be applied to create a diverse set of tasks across different input modalities and configurations. Importantly, this use of indirection enables systematically varying task difficulty through distribution shifts and increasing functional complexity. We conduct a detailed empirical study of different PVR tasks, discovering large variations in performance across dataset sizes, neural network architectures and task complexity. Further, by incorporating distribution shift and increased functional complexity, we develop nuanced tests for reasoning, revealing subtle failures and surprising successes, suggesting many promising directions of exploration on this benchmark.
BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics
Google Research(谷歌研究院);K. Lisa Yang Center for Conservation Bioacoustics(K·丽莎·杨保护生物声学中心);Cornell Lab of Ornithology, Cornell University(康奈尔大学康奈尔鸟类学实验室);Cornell University(康奈尔大学)
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo(滑铁卢大学);Georgia Institute of Technology(佐治亚理工学院);Hong Kong University of Science and Technology(香港科技大学);Google DeepMind(谷歌DeepMind)
STRAP: Spatio-Temporal Pattern Retrieval for Out-of-Distribution Generalization
City University of Hong Kong(香港城市大学);City University of Hong Kong (Dongguan)(香港城市大学(东莞));Northeastern University(东北大学);The Hong Kong Polytechnic University(香港理工大学);University of Electronic Science and Technology of China(电子科技大学);SLAI
Retrieval-augmented Prompt Learning for Pre-trained Foundation Models
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(信息产业部模式分析与机器智能重点实验室,计算机科学与技术学院,南京航空航天大学);Zhejiang University(浙江大学);Hunan Vanguard Group Corporation Limited(湖南先锋集团有限公司);National University of Singapore(新加坡国立大学)