Recent years have seen a growing number of publications that analyse Natural Language Inference (NLI) datasets for superficial cues, whether they undermine the complexity of the tasks underlying those datasets and how they impact those models that are optimised and evaluated on this data. This structured survey provides an overview of the evolving research area by categorising reported weaknesses in models and datasets and the methods proposed to reveal and alleviate those weaknesses for the English language. We summarise and discuss the findings and conclude with a set of recommendations for possible future research directions. We hope it will be a useful resource for researchers who propose new datasets, to have a set of tools to assess the suitability and quality of their data to evaluate various phenomena of interest, as well as those who develop novel architectures, to further understand the implications of their improvements with respect to their model's acquired capabilities.
Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attacks
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute for AI（向量人工智能研究所）；University of Cambridge（剑桥大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）；NSERC CRC Chair（NSERC加拿大研究主席）
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
MIT（麻省理工学院）；Harvard（哈佛大学）；Mass General Brigham（麻省总医院布里格姆医疗系统）；Universitat Pompeu Fabra（庞培法布拉大学）；Aarhus University（奥胡斯大学）；Maastricht University（马斯特里赫特大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；University of Virginia（弗吉尼亚大学）
Latent Performance Profiling of Large Language Models
Department of Electrical Engineering, Indian Institute of Technology Delhi（印度理工学院德里分校电子工程系）；Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi（印度理工学院德里分校人工智能学院）；Hewlett Packard Enterprise, India（印度惠普企业公司）；Department of Computer Science & Engineering, Indian Institute of Technology Kharagpur（印度理工学院Khargapur分校计算机科学与工程系）；A.K.Choudhury School of Information Technology, University of Calcutta, India（印度加尔各答大学信息科技学院）；Department of Computer Science & Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校计算机科学与工程系）；Department of Computer Science, Ashoka University, India（阿什oka大学计算机科学系）；Department of Computer Science & Engineering, Indian Institute of Technology Jodhpur（印度理工学院朱罗普分校计算机科学与工程系）
