The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For \textbf{ability modeling}, we mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering $19$ tasks. (2) For \textbf{data}, to ensure fair comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming to evaluate the capacity to handle unseen data and evolving knowledge. (3) For \textbf{evaluation criteria}, we adopt a contrastive system, including overall standard scores for better numerical comparability across tasks and models and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate $28$ open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset and open-participation leaderboard are publicly released at https://kola.xlore.cn and will be continuously updated to provide references for developing LLMs and knowledge-related systems.
Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室)
RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学)
CANDY: Benchmarking LLMs' Limitations and Assistive Potential in Chinese Misinformation Fact-Checking
School of Cyber Science and Engineering, Sichuan University, China(四川大学网络科学与工程学院);College of Computer Science, Sichuan University, China(四川大学计算机学院);Institute of Data Science, National University of Singapore, Singapore(新加坡国立大学数据科学研究所)
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Lab(上海人工智能实验室);Tsimnghua University(塔密纳大学);Beijing University of Posts and Telecommunications(北京邮电大学);Zhejiang University(浙江大学);University of Tokyo(东京大学)
AD-LLM: Benchmarking Large Language Models for Anomaly Detection
University of Southern California(南加州大学);Northwestern University(西北大学);Arizona State University(亚利桑那州立大学);Adobe Research(Adobe研究);Rice University(里士满大学)
OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases
Southeast University(东南大学);Zhejiang University(浙江大学);Nanjing University of Posts and Telecommunications(南京邮电大学);Tongji University(同济大学);Monash University(莫纳什大学);Soochow University(苏州大学)
Performance of large language models in numerical vs. semantic medical knowledge: Benchmarking on evidence-based Q&As
Tel-Aviv University(特拉维夫大学);Kahun Medical Ltd(Kahun医疗有限公司);Hebrew University of Jerusalem(耶路撒冷希伯来大学);Leumit Health Services(Leumit医疗服务机构);Bar-Ilan University(巴伊兰大学);Kaplan Medical Center(卡普兰医疗中心);Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院)
InnoGym: Benchmarking the Innovation Potential of AI Agents
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);National University of Singapore(新加坡国立大学);Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph(知识图谱联合实验室)
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Alibaba Group(阿里巴巴集团);Beijing Language and Culture University(北京语言大学);LMU Munich(慕尼黑大学);ELLIS Institute Finland(芬兰ELLIS研究所);University of Turku(图尔库大学);IBM Research AI, UAE(阿联酋IBM人工智能研究);MBZUAI;Harbin Institute of Technology, Shenzhen(深圳哈尔滨工业大学);Southern University of Science and Technology(南方科技大学)
Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
Renmin University of China(中国人民大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰霍普金斯大学);Hong Kong University of Science and Technology(香港科技大学);Tencent(腾讯)
KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding
Institute of IT, Kyrgyz State Technical University named after I. Razzakov(以I. 拉扎科夫命名的吉尔吉斯国立技术大学信息技术学院);School of Computer Science, University of Windsor(温莎大学计算机科学学院);St. Petersburg Department of the Steklov Math. Institute, RAS St. Petersburg State University(俄罗斯科学院斯捷克洛夫数学研究所圣彼得堡分部,圣彼得堡国立大学)
From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation
Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences Urumqi China;Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences