State-of-the-art generalisation research in NLP: A taxonomy and review
FAIR；University of Amsterdam（阿姆斯特丹大学）；ETH Zürich（苏黎世联邦理工学院）；University of Edinburgh（爱丁堡大学）；Reka AI；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；University of Cambridge（剑桥大学）；Amazon Alexa AI（亚马逊Alexa AI）；École Normale Supérieure-PSL（巴黎高等师范学院-PSL）；The World Bank（世界银行）；Harvard University（哈佛大学）；University of Aberdeen（阿伯丁大学）；IT University of Copenhagen（哥本哈根信息技术大学）；Pioneer Centre for Artificial Intelligence（先锋人工智能中心）；Textshuttle；National University of Mongolia（蒙古国立大学）；Hong Kong University of Science and Technology（香港科技大学）；MIT（麻省理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, Zhijing Jin
备注：
This preprint was published as an Analysis article in Nature Machine Intelligence. Please refer to the published version when citing this work. 28 pages of content + 6 pages of appendix + 52 pages of references
The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not well understood, nor are there any evaluation standards for generalisation. In this paper, we lay the groundwork to address both of these issues. We present a taxonomy for characterising and understanding generalisation research in NLP. Our taxonomy is based on an extensive literature review of generalisation research, and contains five axes along which studies can differ: their main motivation, the type of generalisation they investigate, the type of data shift they consider, the source of this data shift, and the locus of the shift within the modelling pipeline. We use our taxonomy to classify over 400 papers that test generalisation, for a total of more than 600 individual experiments. Considering the results of this review, we present an in-depth analysis that maps out the current state of generalisation research in NLP, and we make recommendations for which areas might deserve attention in the future. Along with this paper, we release a webpage where the results of our review can be dynamically explored, and which we intend to update as new NLP generalisation studies are published. With this work, we aim to take steps towards making state-of-the-art generalisation testing the new status quo in NLP.
School of Computer Science（计算机科学学院）；Engineering, Central South University（工程学院，中南大学）；Research Center for Social Computing（社会计算研究中心）；Interactive Robotics, Harbin Institute of Technology（交互机器人研究，哈尔滨工业大学）；Department of Computer Science, University of Hong Kong（计算机科学系，香港大学）；Shenzhen International Graduate School, Tsinghua University（深圳国际研究生院，清华大学）；Department of Computer Science, University of Illinons at Chicago（计算机科学系，伊利诺伊大学香槟分校）
How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond
Sichuan University（四川大学）；Singapore Management University（新加坡国立大学）；York University（约克大学）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China（机器学习与产业智能工程研究中心，教育部，中国）；National University of Singapore（新加坡国立大学）
When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
School of Law, China University of Political Science and Law（中国政法大学法学院）；Zhejiang University of Finance and Economics Dongfang College（浙江财经大学东洋学院）；School of Information Management for Law, China University of Political Science and Law（中国政法大学信息管理法学院）；Department of Artificial Intelligence, Chung-Ang University（成均馆大学人工智能系）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（香港理工大学数据科学与人工智能系）
Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art
Technical University of Darmstadt（达姆施塔特工业大学）；Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)（康拉德·楚泽学习与智能系统卓越学院（ELIZA））；University of Cambridge（剑桥大学）
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Massachusetts Institute of Technology（麻省理工学院）；Allen Institute for AI（艾伦人工智能研究所）
Achieving Peak Performance for Large Language Models: A Systematic Review
Óbuda University（奥布达大学）；J. Selye University（塞耶大学）；Institute for Computer Science and Control (SZTAKI)（计算机科学与控制研究所（SZTAKI））；Hungarian Research Network (HUN-REN)（匈牙利研究网络（HUN-REN））
Decoding the Diversity: A Review of the Indic AI Research Landscape
AI Institute, University of South Carolina（南卡罗来纳大学AI学院）；University of South Carolina（南卡罗来纳大学）；Stanford University（斯坦福大学）；DeepSig Inc.（DeepSig公司）；Amazon GenAI（亚马逊生成式AI部门）
The Nature of NLP: Analyzing Contributions in NLP Papers
Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technische Universität Darmstadt（德累斯顿技术大学计算机科学系及海瑟森人工智能中心）；IT:U Interdisciplinary Transformation University Austria（奥地利 interdisciplinary Transformation 大学）；IBM Research Europe - Ireland（IBM欧洲爱尔兰研究中心）；National Research Council Canada（加拿大国家研究委员会）
COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews
Engg. Indian Institute of Technology, Gandhinagar Gujarat, India singh\；Engg. Indian Institute of Technology, Gandhinagar Gujarat, India；Engg. Indian Institute of Technology, Kharagpur West Bengal, India
