Large Language Models (LLMs) are increasingly deployed in safety-critical systems where they provide answers based on in-context information derived from knowledge bases. As LLMs are increasingly envisioned as superhuman agents, their proficiency in knowledge comprehension-extracting relevant information and reasoning over it to answer questions, a key facet of human intelligence-becomes crucial. However, existing evaluations of LLMs on knowledge comprehension are typically conducted on small test sets, but these datasets represent only a tiny fraction of the vast number of possible queries. Simple empirical evaluations on these limited test sets raises concerns about the reliability and generalizability of the results. In this work, we introduce the first specification and certification framework for knowledge comprehension in LLMs, providing formal probabilistic guarantees for reliability. Instead of a fixed dataset, we design novel specifications that mathematically represent prohibitively large probability distributions of knowledge comprehension prompts with natural noise, using knowledge graphs. From these specifications, we generate quantitative certificates that offer high-confidence, tight bounds on the probability that a given LLM correctly answers any question drawn from the specification distribution. We apply our framework to certify SOTA LLMs in two domains: precision medicine and general question-answering. Our results reveal previously unrecognized vulnerabilities in SOTA LLMs due to natural noise in the prompts. Additionally, we establish performance hierarchies with formal guarantees among the SOTA LLMs, particularly in the context of precision medicine question-answering.
Knowlesdge Lab, University of Chicago, Chicago, Illinois, USA（知识实验室，芝加哥大学，芝加哥，伊利诺伊州，美国）；Knowledge Lab, Data Science Institute, Department of Sociology, University of Chicago, Chicago, Illinois, USA（知识实验室，数据科学学院，社会学系，芝加哥大学，芝加哥，伊利诺伊州，美国）
SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View
Sichuan University（四川大学）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education（教育部机器学习与工业智能工程研究中心）；School of Statistics and Data Science, AAIS, Nankai University（南开大学统计与数据科学学院）
Generating Reading Comprehension Exercises with Large Language Models for Educational Applications
School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China（光学电子与计算机工程学院，上海理工大学，上海，中国）；Chongqing Academy of Science and Technology, Chongqing, China（重庆市科学技术院，重庆，中国）
Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）；Georgia Institute of Technology（佐治亚理工学院）；Stanford University（斯坦福大学）；Amazon AI（亚马逊人工智能）
How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehension
Shenzhen Graduate School, Peking University（北京大学深圳研究生院）；International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；Pengcheng Laboratory（鹏城实验室）；Hong Kong University of Science and Technology（香港科技大学）
TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Text
School of Mathematical Sciences（数学科学学院）；Tel Aviv University（特拉维夫大学）；Department of Electrical Engineering（电气工程系）；The Faculty of Engineering（工程学院）；Bar Ilan University（巴伊兰大学）
When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models
Blavatnik School of Computer Science, Tel Aviv University（特拉维夫大学布拉瓦尼克计算机科学学院）；Department of Linguistics, Tel Aviv University（特拉维夫大学语言学系）；Sagol School of Neuroscience, Tel Aviv University（特拉维夫大学萨戈尔神经科学学院）
Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea（韩国立科学技术院人工智能研究生院）；Department of Computer Science and Engineering, POSTECH, Republic of Korea（韩国立科学技术院计算机科学与工程系）
SLANG: New Concept Comprehension of Large Language Models
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of California, Merced（加利福尼亚大学默塞德分校）；University of Chinese Academy of Sciences（中国科学院大学）
ASMA-Tune: Unlocking LLMs' Assembly Code Comprehension via Structural-Semantic Instruction Tuning
Nankai University（南开大学）；Zhejiang University（浙江大学）；Ant Group（蚂蚁集团）；Renmin University of China（中国人民大学）；University of the Chinese Academy of Sciences（中国科学院大学）
Text Production and Comprehension by Human and Artificial Intelligence: Interdisciplinary Workshop Report
Nottingham Trent University（诺丁汉特伦特大学）；Cornell University（康奈尔大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of the Saarland（萨尔大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Edinburgh（爱丁堡大学）；Tuebingen University（图宾根大学）；Montclair State University（蒙特克莱尔州立大学）；Carnegie Mellon University（卡内基梅隆大学）；Georgia Institute of Technology（佐治亚理工学院）；Iowa State University（爱荷华州立大学）；Embry-Riddle Aeronautical University（安柏瑞德航空航天大学）；Midland University（米德兰大学）；University of Chicago（芝加哥大学）；University of Iowa（爱荷华大学）
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Maryland（马里兰大学）；Virginia Tech（弗吉尼亚理工大学）
