Large Language Models (LLMs) can revolutionize how we deploy and operate Open Radio Access Networks (O-RAN) by enhancing network analytics, anomaly detection, and code generation and significantly increasing the efficiency and reliability of a plethora of O-RAN tasks. In this paper, we present ORAN-Bench-13K, the first comprehensive benchmark designed to evaluate the performance of Large Language Models (LLMs) within the context of O-RAN. Our benchmark consists of 13,952 meticulously curated multiple-choice questions generated from 116 O-RAN specification documents. We leverage a novel three-stage LLM framework, and the questions are categorized into three distinct difficulties to cover a wide spectrum of ORAN-related knowledge. We thoroughly evaluate the performance of several state-of-the-art LLMs, including Gemini, Chat-GPT, and Mistral. Additionally, we propose ORANSight, a Retrieval-Augmented Generation (RAG)-based pipeline that demonstrates superior performance on ORAN-Bench-13K compared to other tested closed-source models. Our findings indicate that current popular LLM models are not proficient in O-RAN, highlighting the need for specialized models. We observed a noticeable performance improvement when incorporating the RAG-based ORANSight pipeline, with a Macro Accuracy of 0.784 and a Weighted Accuracy of 0.776, which was on average 21.55% and 22.59% better than the other tested LLMs.
Plant in Cupboard, Orange on Rably, Inat Aphone. Benchmarking Incremental Learning of Situation and Language Model using a Text-Simulated Situated Environment
Computational Linguistics, Department of Linguistics University of Potsdam（语言学系计算语言学部门波恩大学）；German Research Center for Artificial Intelligence (DFKI), Berlin（德国人工智能研究中心（DFKI）柏林）
Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports
Indian Institute of Technology Patna（印度理工学院帕纳布分校）；Sardar Patel Institute of Technology（萨达尔·帕特尔技术学院）；Universitas Gadjah Mada（加查马大学）；King Mongkut’s Institute of Technology Ladkrabang（拉差班国王技术学院）；Shenzhen Technology University（深圳技术大学）；Université de Toulouse（图卢兹大学）
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing Future Brain Education Technology Co., Ltd.（北京未来脑教育科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Guangxi Zhuang Autonomous Region Big Data Research Institute（广西壮族自治区大数据研究院）
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine（上海中医药大学针灸推拿学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；China Academy of Chinese Medical Sciences（中国中医科学院）
Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation
AI Singapore（新加坡人工智能中心）；SCB 10X；School of Information Science and Technology, VISTEC（VISTEC信息科学与技术学院）；Chulalongkorn University（朱拉隆功大学）；National University of Singapore（新加坡国立大学）
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University（沃克生物医学工程部门，佐治亚理工学院和埃默里大学）；College of Future of Technology, Peking University（未来技术学院，北京大学）；School of Architecture, Tsinghua University（建筑学院，清华大学）；School of Electrical and Computer Engineering, Georgia Institute of Technology（电气与计算机工程学院，佐治亚理工学院）；School of Computer Science, Georgia Institute of Technology（计算机科学学院，佐治亚理工学院）
Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements
Provable Responsible AI and Data Analytics (PRADA) Lab（可证责任AI与数据分析实验室）；King Abdullah University of Science and Technology（卡瓦尔大学科学与技术大学）；University of Toronto（多伦多大学）；University of Macau（澳门大学）；University of Electronic Science and Technology of China（电子科技大学）
ElectriQ: A Benchmark for Assessing the Response Capability of Large Language Models in Power Marketing
Systems Engineering Institute, Xi’an Jiaotong University（系统工程研究所，西安交通大学）；State Key Laboratory of Multiphase Flow in Power Engineering, School of Energy and Power Engineering, Xi’an Jiaotong University（电力工程多相流国家重点实验室，能源与动力工程学院，西安交通大学）；School of Electronic Science and Engineering, Xi'an Jiaotong University（电子科学与工程学院，西安交通大学）；Organizational Management Department, School of Management, Xi’an Jiaotong University（组织管理部，管理学院，西安交通大学）
Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks
University of Oxford（牛津大学）；Microsoft Research（微软研究院）；The Alan Turing Institute（艾伦·图灵研究所）；Allen Institute for AI（人工智能研究所）；University of Washington（华盛顿大学）；Microsoft Corporation（微软公司）；University of York（约克大学）
PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Models
King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；University of Western Australia（西澳大学）；Ontario Tech University（安大略理工大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）
Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs
Massachusetts Institute of Technology（麻省理工学院）；University of Michigan - Ann Arbor（密歇根大学安娜堡分校）；National Yang Ming Chiao Tung University（国立阳明交通大学）；Harvard University（哈佛大学）；University of the Philippines（菲律宾大学）；Stevens Institute of Technology（史蒂文斯理工学院）
