RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
University of Edinburgh（爱丁堡大学）；HSE University（高等经济大学）；University of Groningen（格罗宁根大学）；Ghent University（根特大学）；SaluteDevices（SaluteDevices公司）；Toloka AI（Toloka AI公司）；University of Oslo（奥斯陆大学）
最小对立体是评估语言模型语法知识的一种成熟方法。然而，现有的最小对立体资源覆盖的语言数量有限，并且缺乏针对特定语言的语法现象多样性。本文介绍了俄语语言最小对立体基准（Russian Benchmark of Linguistic Minimal Pairs，RuBLiMP），其中包含4.5万对在语法合格性上存在差异的句子，并分别隔离出某一形态、句法或语义现象。与现有语言最小对立体基准不同，RuBLiMP通过对来自开放文本语料库的自动标注句子施加语言扰动，并精心筛选测试数据而构建。我们描述了数据收集流程，并展示了在多种场景下评估25个语言模型的结果。我们发现，广泛使用的俄语语言模型对形态以及与一致关系相关的对比较为敏感，但在需要理解结构关系、否定、及物性和时态的现象上落后于人类。RuBLiMP、代码库及其他材料均已公开发布。
英文摘要：
Minimal pairs are a well-established approach to evaluating the grammatical knowledge of language models. However, existing resources for minimal pairs address a limited number of languages and lack diversity of language-specific grammatical phenomena. This paper introduces the Russian Benchmark of Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon. In contrast to existing benchmarks of linguistic minimal pairs, RuBLiMP is created by applying linguistic perturbations to automatically annotated sentences from open text corpora and carefully curating test data. We describe the data collection protocol and present the results of evaluating 25 language models in various scenarios. We find that the widely used language models for Russian are sensitive to morphological and agreement-oriented contrasts but fall behind humans on phenomena requiring understanding of structural relations, negation, transitivity, and tense. RuBLiMP, the codebase, and other materials are publicly available.
RuCoLA: Russian Corpus of Linguistic Acceptability
SberDevices；ABBYY；HSE University（高等经济大学）；Yandex；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Center for Information and Language Processing (CIS), MaiNLP lab, LMU Munich（慕尼黑大学信息与语言处理中心（CIS）MaiNLP实验室）
Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Vector Institute of AI（Vector人工智能研究所）；University of Central Florida（中佛罗里达大学）；University of British Columbia（不列颠哥伦比亚大学）；Emory University（埃默里大学）；Invertible AI
Zero-shot Cross-Linguistic Learning of Event Semantics
Indiana University（印第安纳大学）；Rutgers University（罗格斯大学）；University of Pittsburgh（匹兹堡大学）；University of Edinburgh（爱丁堡大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）
Friend or Foe? Language as an ideological switch in open-weight LLMs under Russian disinformation stress
Institute of Culture Studies, University of Silesia in Katowice（文化研究学院，卡托维察大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；National Aviation University（国家航空大学）
MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Technical University of Berlin（柏林工业大学）
Grammatical cues to subjecthood are redundant in a majority of simple clauses across languages
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Vanderbilt University（范德堡大学）；Massachusetts Institute of Technology（麻省理工学院）；University of California, Irvine（加州大学欧文分校）
Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models
Research Center Trustworthy Data Science and Security of the University Alliance Ruhr（鲁尔大学联盟可信数据科学与安全研究中心）；Faculty of Informatics, Technical University Dortmund（多特蒙德工业大学信息学院）
A Culturally-diverse Multilingual Multimodal Video Benchmark & Model
Mohamed bin Zayed University of Artificial Intelligence（莫德赫·本·扎耶德人工智能大学）；University of Central Florida（中央佛罗里达大学）；Islamic University of Technology（伊斯兰技术大学）；Air University（空军大学）；ETH Zurich（苏黎世联邦理工学院）；Technische Universität München（慕尼黑技术大学）；National Institute of Informatics（国家信息研究所）；Australian National University（澳大利亚国立大学）；Linköping University（利尔贝里大学）
