A Principled Framework for Evaluating on Typologically Diverse Languages
Aalborg University（奥胡斯大学）；KU Leuven（库勒芬大学）；Aalborg University Department of Computer Science（奥胡斯大学计算机科学系）；KU Leuven Department of Computer Science（库勒芬大学计算机科学系）
Esther Ploeger, Wessel Poelman, Andreas Holck Høeg-Petersen, Anders Schlichtkrull, Miryam de Lhoneux, Johannes Bjerva
Beyond individual languages, multilingual natural language processing (NLP) research increasingly aims to develop models that perform well across languages generally. However, evaluating these systems on all the world's languages is practically infeasible. To attain generalizability, representative language sampling is essential. Previous work argues that generalizable multilingual evaluation sets should contain languages with diverse typological properties. However, 'typologically diverse' language samples have been found to vary considerably in this regard, and popular sampling methods are flawed and inconsistent. We present a language sampling framework for selecting highly typologically diverse languages given a sampling frame, informed by language typology. We compare sampling methods with a range of metrics and find that our systematic methods consistently retrieve more typologically diverse language selections than previous methods in NLP. Moreover, we provide evidence that this affects generalizability in multilingual model evaluation, emphasizing the importance of diverse language sampling in NLP evaluation.
MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages
University of Washington（华盛顿大学）；Massachusetts Institute of Technology（麻省理工学院）；Penn State University（宾夕法尼亚州立大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Studio Ousia（Ousia工作室）；RIKEN（日本理化学研究所）；Google Research（谷歌研究院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages
Saarland University（萨尔大学）；University of Pretoria（比勒陀利亚大学）；Bayero University Kano（贝罗大学卡诺分校）；Technical University of Munich（慕尼黑工业大学）；Lanfrica；Lelapa AI；SADiLaR（南非语言资源中心）；Malawi University of Business and Applied Science（马拉维商业与应用科学大学）；Nnamdi Azikiwe University（纳姆迪·阿齐基韦大学）；University of Buea（布埃亚大学）；University of Ibadan（伊巴丹大学）；Ewegbe Akademi（Ewegbe学院）；AMALAN；University of Porto（波尔图大学）
Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?
Sway AI（Sway人工智能公司）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Toronto（多伦多大学）；University of Moratuwa（莫拉图瓦大学）；IIT(BHU) Varanasi（瓦拉纳西印度理工学院（班纳拉斯印度大学））；Saarland University（萨尔大学）
Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages
Imperial College London（伦敦帝国学院）；University of South Africa（南非大学）；University of Johannesburg（约翰内斯堡大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席项目）；Masakhane General Purpose（Masakhane通用项目）；Saarland University（萨尔大学）；Equiano Institute（埃夸诺研究所）；OpenAI（OpenAI公司）
EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Singapore Institute of Manufacturing Technology, A*STAR（新加坡制造技术研究所，A*STAR）
From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment
Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Lund University（吕勒奥大学）；University of Tübingen（图宾根大学）；Stanford Graduate School of Education（斯坦福大学教育研究生院）；Harvard Graduate School of Education（哈佛大学教育研究生院）
Evaluating and Inducing Personality in Pre-trained Language Models
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Yuanpei College, Peking University（北京大学元培学院）；National Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院通用人工智能全国重点实验室）；Beijing Jiaotong University（北京交通大学）
