GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）；Central South University（中南大学）；University of Electronic Science and Technology of China（电子科技大学）；Microsoft Research Asia（微软亚洲研究院）
Pre-trained language models (PLMs) are known to improve the generalization performance of natural language understanding models by leveraging large amounts of data during the pre-training phase. However, the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks, limiting the real-world deployment of these methods. This paper presents the first attempt at creating a unified benchmark named GLUE-X for evaluating OOD robustness in NLP models, highlighting the importance of OOD robustness and providing insights on how to measure the robustness of a model and how to improve it. The benchmark includes 13 publicly available datasets for OOD testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly used PLMs, including GPT-3 and GPT-3.5. Our findings confirm the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy.
GLUECons: A Generic Benchmark for Learning Under Constraints
Michigan State University（密歇根州立大学）；Florida Institute for Human and Machine Cognition（佛罗里达人机认知研究所）；University of Pennsylvania（宾夕法尼亚大学）；University of California Berkeley（加州大学伯克利分校）
Evaluating Large Language Model Creativity from a Literary Perspective
Imperial College London（帝国理工学院）；Google DeepMind（谷歌DeepMind）；Institute of Historical Research, School of Advanced Study, University of London（伦敦大学高级研究院历史研究所）；University of Southampton（南安普顿大学）
A Multi-Perspective Benchmark and Moderation Model for Evaluating Safety and Adversarial Robustness
Department of Computer and Information Science, Fordham University（福德汉大学计算机与信息科学系）；College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·卡西姆大学科学与工程学院）；Department of Computer Science and Engineering, Qatar University（卡塔尔大学计算机科学与工程系）
Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室，计算技术研究所）；Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院网络数据科学与技术重点实验室，计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Models
Information Technology University Department of Computer Science（信息科技大学计算机科学系）；Zayed University College of Interdisciplinary Studies（扎耶德大学跨学科研究学院）；Computer Science and Engineering Department（计算机科学与工程系）
DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective
State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Nanyang Technological University（南洋理工大学）；Chongqing University（重庆大学）；University of Maryland at College Park（马里兰大学学院公园分校）
Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective
Fudan University（复旦大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhejiang University（浙江大学）；Renmin University of China（中国人民大学）；Alibaba Group（阿里巴巴集团）
CLIP-DQA: Blindly Evaluating Dehazed Images from Global and Local Perspectives Using CLIP
School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；Alibaba Group（阿里巴巴集团）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；BNRist, Tsinghua University（清华大学BNRist）
How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
Harbin Institute of Technology（哈尔滨工业大学）；Central South University（中南大学）；Chinese Academy of Sciences（中国科学院）；Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models
Department of Computer Science, University of Southern California（计算机科学系，南加州大学）；Information Sciences Institute, University of Southern California（信息科学研究所，南加州大学）
TriWorldBench: A Tri-View Consistency Perspective on Embodied World Models
Peking University（北京大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；DeepMind（深度思维）
EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学至理书院）
SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions
The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）；Johns Hopkins University（约翰霍普金斯大学）；University of California Irvine（加州大学尔湾分校）；Stanford University（斯坦福大学）
Consolidating and Developing Benchmarking Datasets for the Nepali Natural Language Understanding Tasks
Information and Language Processing Research Lab (ILPRL)（信息与语言处理研究实验室）；Kathmandu University（加德满都大学）；Institute for Research and Innovation in Intelligent Systems (IRIIS)（智能系统研究与创新研究院）
Pre-training Language Model as a Multi-perspective Course Learner
Microsoft Corporation（微软公司）；National Engineering Research Center of Speech and Language Information Processing（国家语音及语言信息处理工程技术研究中心）；University of Science and Technology of China（中国科学技术大学）
