The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis
Saarland University（萨尔大学）；Bosch Center for AI（博世人工智能中心）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Eastern Institute of Technology（东方理工学院）；Mila（米拉研究所）；McGill University（麦吉尔大学）
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba O. Alabi, Xiaoyu Shen, Dietrich Klakow, Marius Mosbach
In-context learning is a popular inference strategy where large language models solve a task using only a few labeled demonstrations without needing any parameter updates. Although there have been extensive studies on English in-context learning, multilingual in-context learning remains under-explored, and we lack an in-depth understanding of the role of demonstrations in this context. To address this gap, we conduct a multidimensional analysis of multilingual in-context learning, experimenting with 5 models from different model families, 9 datasets covering classification and generation tasks, and 56 typologically diverse languages. Our results reveal that the effectiveness of demonstrations varies significantly across models, tasks, and languages. We also find that strong instruction-following models including Llama 2-Chat, GPT-3.5, and GPT-4 are largely insensitive to the quality of demonstrations. Instead, a carefully crafted template often eliminates the benefits of demonstrations for some tasks and languages altogether. These findings show that the importance of demonstrations might be overestimated. Our work highlights the need for granular evaluation across multiple axes towards a better understanding of in-context learning.
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Alibaba Group（阿里巴巴集团）；TU Wien（维也纳技术大学）；University of Chinese Academy of Sciences（中国科学院大学）
A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluation
New York University（纽约大学）；University of South Florida（佛罗里达州立大学）；Google LLC（谷歌公司）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；Massachusetts Institute of Technology（麻省理工学院）
LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
China-ASEAN Information Harbor Co., Ltd.（中国-东盟信息港有限公司）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；School of Software & Microelectronics, Peking University（北京大学软件与微电子学院）
DynamicNER: A Dynamic, Multilingual, and Fine-Grained Dataset for LLM-based Named Entity Recognition
New York University Abu Dhabi（纽约大学阿布扎赫德分校）；Zhejiang University（浙江大学）；The Hong Kong Polytechnic University（香港理工大学）；Nanyang Technology University（南阳技术大学）；University of Electronic Science and Technology of China（电子科技大学）；Texas A&M University（德克萨斯大学）；Squirrel AI
Open-Source Large Language Models as Multilingual Crowdworkers: Synthesizing Open-Domain Dialogues in Several Languages With No Examples in Targets and No Machine Translation
IKS-Instruct: A 24,000-Example Multilingual Dataset for Teaching Language Models Indian Knowledge Systems
Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi（印度理工学院德里分校传感器、仪器仪表和网络物理系统工程中心（传感中心）本征实验室）；RSL Quantum, FITT, IIT Delhi（印度理工学院德里分校 FITT 学院 RSL 量子实验室）
University of North Texas（北得克萨斯大学）；National Research Council（国家研究委员会）
Development of language proficiency models for non-native learners has been an active area of interest in NLP research for the past few years. Although language proficiency is multidimensional in nature, existing research typically considers a single "overall proficiency" while building models. Further, existing approaches also considers only one language at a time. This paper describes our experiments and observations about the role of pre-trained and fine-tuned multilingual embeddings in performing multi-dimensional, multilingual language proficiency classification. We report experiments wit…
Automatic hate speech detection in online social networks is an important open problem in Natural Language Processing (NLP). Hate speech is a multidimensional issue, strongly dependant on language and cultural factors. Despite its relevance, research on this topic has been almost exclusively devoted to English. Most supervised learning resources, such as labeled datasets and NLP tools, have been created for this same language. Considering that a large portion of users worldwide speak in languages other than English, there is an important need for creating efficient approaches for multilingual …
We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations. We also provid…
Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their training data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train multilingual generative language models on a corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters sets new state of the art in few-shot learning in more than 20 representative lang…
N2C2: Nearest Neighbor Enhanced Confidence Calibration for Cross-Lingual In-Context Learning
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human…
UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment
University of Bath（巴斯大学）；Cardiff University（卡迪夫大学）；National University Philippines（菲律宾国家大学）；Bielefeld University（比勒菲尔德大学）；University of Exeter（埃克塞特大学）；University of Gothenburg（哥德堡大学）；UCLouvain（列日大学）；MBZUAI（马克斯·普朗克智能系统研究所）；Brigham Young University（Brigham Young 大学）；INESC-ID Lisboa（里斯本 INESC-ID）；Instituto Universitário de Lisboa (ISCTE-IUL)（里斯本大学学院（ISCTE-IUL））；Universitat Pompeu Fabra（庞培法华大学）；National Research Council, Canada（加拿大国家研究委员会）；King Abdulaziz University（国王阿卜杜勒-阿齐兹大学）
When Less Is Enough: Context Selection and Prompting Strategies for Bengali News Headline Generation
Charles Sturt University（查尔斯斯特大学）；Chittagong University of Engineering and Technology（吉大港工程技术大学）；Bangladesh Army University of Science and Technology（孟加拉国陆军理工大学）
AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages
University College London（伦敦大学学院）；Masakhane NLP（Masakhane自然语言处理组织）；University of Maryland（马里兰大学）；University of Oxford（牛津大学）；Unbabel（Unbabel公司）；Instituto Superior Técnico（高等技术学院）；INESC-ID（INESC-ID研究所）；ENSIAS（国家高等计算机与系统分析学院）；SADiLaR（南非洲数字语言资源中心）；Aston University（阿斯顿大学）；University of Eastern Finland（东芬兰大学）；Luleå University of Technology（吕勒奥理工大学）；Fudan University（复旦大学）；Conservatoire National des Arts et Métiers（法国国立工艺学院）；Lelapa AI（Lelapa人工智能公司）；Imperial College London（帝国理工学院）；University of Deusto（德乌斯托大学）
