In humans, Attention is a core property of all perceptual and cognitive operations. Given our limited ability to process competing sources, attention mechanisms select, modulate, and focus on the information most relevant to behavior. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. For the last six years, this property has been widely explored in deep neural networks. Currently, the state-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models. We systematically reviewed hundreds of architectures in the area, identifying and discussing those in which attention has shown a significant impact. We also developed and made public an automated methodology to facilitate the development of reviews in the area. By critically analyzing 650 works, we describe the primary uses of attention in convolutional, recurrent networks and generative models, identifying common subgroups of uses and applications. Furthermore, we describe the impact of attention in different application domains and their impact on neural networks' interpretability. Finally, we list possible trends and opportunities for further research, hoping that this review will provide a succinct overview of the main attentional models in the area and guide researchers in developing future approaches that will drive further improvements.
Visual Attention Methods in Deep Learning: An In-Depth Survey
University of South Australia（南澳大学）；Fayoum University（法尤姆大学）；King Fahad University of Petroleum and Minerals（法赫德国王石油与矿业大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）；University of Canberra（堪培拉大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Linkoping University（林雪平大学）；University of Western Australia（西澳大利亚大学）
School of Computer Science and Technology, China University of Mining and Technology（中国矿业大学计算机科学与技术学院）；Mine Digitization Engineering Research Center, Ministry of Education of the People’s Republic of China（教育部矿山数字化工程研究中心）
Survey on Generalization Theory for Graph Neural Networks
RWTH Aachen University（亚琛工业大学）；Technical University of Munich（慕尼黑工业大学）；Massachusetts Institute of Technology（麻省理工学院）；Technion - Israel Institute of Technology（以色列理工学院）
School of Artificial Intelligence, Nanjing University of Information Science and Technology（南京信息工程大学人工智能学院）；Tsinghua University（清华大学）；Institute of Cyberspace Security and College of Information Engineering, Zhejiang University of Technology（浙江工业大学网络空间安全研究院与信息工程学院）；University of Edinburgh（爱丁堡大学）；Beijing Institute of Technology（北京理工大学）；College of Computer Science and Engineering, Dalian Minzu University（大连民族大学计算机科学与工程学院）；Durham University（杜伦大学）；Northumbria University（诺森比亚大学）
A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention
INESC TEC - Institute for Systems and Computer Engineering, Technology and Science（INESC TEC系统与计算机工程、技术与科学研究所）；University of Porto - Faculty of Engineering（波尔图大学工程学院）；IMP Diagnostics（IMP诊断公司）；Portuguese Oncology Institute of Porto (IPO Porto)（波尔图葡萄牙肿瘤研究所）；University of Porto (ICBAS-UP)（波尔图大学（ICBAS-UP））；FCTUC - Faculty of Science and Technology, University of Coimbra（科英布拉大学科学与技术学院（FCTUC））
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Tsinghua University（清华大学）；Meituan LongCat Team（美团LongCat团队）；The University of Hong Kong（香港大学）；University of Michigan（密歇根大学）；Xiamen University（厦门大学）；The Ohio State University（俄亥俄州立大学）；Columbia University（哥伦比亚大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）
Peer Disambiguation in Self-Reported Surveys using Graph Attention Networks
University of Southern California（南加州大学）；Ming Hsieh Department of Electrical and Computer Engineering（明赫氏电气与计算机工程学院）；Department of Computer Science（计算机科学系）；Suzanne Dworak-Peck School of Social Work（苏珊娜·德沃拉克-佩克社会工作学院）
