Despite the remarkable advancements and widespread applications of deep neural networks, their ability to perform reasoning tasks remains limited, particularly in domains requiring structured, abstract thought. In this paper, we investigate the linguistic reasoning capabilities of state-of-the-art large language models (LLMs) by introducing IOLBENCH, a novel benchmark derived from International Linguistics Olympiad (IOL) problems. This dataset encompasses diverse problems testing syntax, morphology, phonology, and semantics, all carefully designed to be self-contained and independent of external knowledge. These tasks challenge models to engage in metacognitive linguistic reasoning, requiring the deduction of linguistic rules and patterns from minimal examples. Through extensive benchmarking of leading LLMs, we find that even the most advanced models struggle to handle the intricacies of linguistic complexity, particularly in areas demanding compositional generalization and rule abstraction. Our analysis highlights both the strengths and persistent limitations of current models in linguistic problem-solving, offering valuable insights into their reasoning capabilities. By introducing IOLBENCH, we aim to foster further research into developing models capable of human-like reasoning, with broader implications for the fields of computational linguistics and artificial intelligence.
Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali
Dept. of Computer Science & Engineering（计算机科学与工程系）；Nepal Engineering College（尼泊尔工程学院）；Central Dept. of CS and IT（计算机科学与信息科技中央系）；Tribhuvan University（特里布文大学）
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能学院）；Rochester Institute of Technology（罗切斯特理工学院）；Meta AI；Amazon Gen AI（亚马逊生成式人工智能部门）
The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning
University College London（伦敦大学学院）；Meta；CentraleSupélec（中央高等电力学院）；McGill University（麦吉尔大学）；Cohere Labs（Cohere实验室）；Princeton University（普林斯顿大学）；University of Amsterdam（阿姆斯特丹大学）；Stockholm University（斯德哥尔摩大学）；Faculty of Liberal Arts and Sciences（文理学院）；Universidade Federal de Goiás（戈亚斯联邦大学）
MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
Harvard University（哈佛大学）；MIT（麻省理工学院）；Cornell University（康奈尔大学）；Mayo Clinic（梅奥诊所）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；University of Virginia（弗吉尼亚大学）；Harvard Medical School（哈佛医学院）；Abaka AI；NYU（纽约大学）
Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR
Chengdu Institute of Computer Applications, CAS（成都计算机应用研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；IMRAM, Tohoku University（东大理工学院IMRAM）；China Tower Corporation Limited（中国塔公司）；CSIS, Tohoku University（东大理工学院CSIS）；National Institute for Materials Science（国家材料科学研究所）；Shenzhen Institutes of Advanced Technology, CAS（深圳先进技术研究所，中国科学院）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术大学人工智能研究院）
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））；Peking University（北京大学）；Mindverse AI；Alibaba Group（阿里巴巴集团）；Zhejiang University（浙江大学）；Zhejiang University of Technology（浙江工业大学）；MBZUAI；NIO（蔚来汽车）；HSBC（汇丰银行）；Harvard T.H. Chan School of Public Health（哈佛大学T.H. Chan公共卫生学院）
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models
University of Helsinki（赫尔辛基大学）；Technical University of Darmstadt（达姆施塔特技术大学）；University of Munich（慕尼黑大学）；Nanjing University（南京大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs
Provable Responsible AI and Data Analytics (PRADA) Lab（可证负责任人工智能与数据 analytics 实验室）；King Abdullah University of Science and Technology（国王阿卜杜勒阿齐兹科学与技术大学）；HKUST（香港科技大学）；MBZUAI（穆罕默德·本·拉希德人工智能研究所）；University of Science and Technology of China（中国科学技术大学）；Kyungpook National University（庆尚国立大学）
Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Models
Computational Linguistics, Department of Linguistics University of Potsdam（波恩大学语言学系）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks
Center for Language Engineering（语言工程中心）；Al-Khawarizmi Institute of Computer Science（阿尔-花剌子密计算机科学研究所）；University of Engineering and Technology Lahore（拉合尔工程技术大学）
A Large-Scale Chinese Knowledge Graph-Text Alignment Dataset for Benchmarking Knowledge-Grounded LLMs
Nanjing University of Science and Techonolgy（南京理工大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Science and Technology Beijing（北京科技大学）；Hefei University of Technology（合肥工业大学）；Beijing University of Chemical Technology（北京化工大学）；Renmin University of China（中国人民大学）；Beihang University（北航）；Beijing University of Posts and Telecommunications（北京邮电大学）；Griffith University, Australia（澳大利亚格里菲斯大学）
XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad
Department of Computer Science, National Centre for Text Mining, The University of Manchester（计算机科学系，国家文本挖掘中心，曼彻斯特大学）；ELLIS Manchester（曼彻斯特ELLIS）；School of Computing, Queen’s University, Ontario, Canada（计算学院，加拿大皇后大学）；Computer Science, University of Illinois Chicago（计算机科学，伊利诺伊大学芝加哥分校）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；Department of Computer Science and Artificial Intelligence, Umm Al-Qura University, Makkah, Saudi Arabia（计算机科学与人工智能系，乌姆·阿勒·卡拉大学，麦加，沙特阿拉伯）
