Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering
Institute for Applied Informatics（应用信息学研究所）；Leipzig University（莱比锡大学）
Lars-Peter Meyer, Johannes Frey, Kurt Junghanns, Felix Brei, Kirill Bulert, Sabine Gründer-Fahrer, Michael Martin
备注：
To be published in SEMANTICS 2023 poster track proceedings. SEMANTICS 2023 EU: 19th International Conference on Semantic Systems, September 20-22, 2023, Leipzig, Germany
As the field of Large Language Models (LLMs) evolves at an accelerated pace, the critical need to assess and monitor their performance emerges. We introduce a benchmarking framework focused on knowledge graph engineering (KGE) accompanied by three challenges addressing syntax and error correction, facts extraction and dataset generation. We show that while being a useful tool, LLMs are yet unfit to assist in knowledge graph generation with zero-shot prompting. Consequently, our LLM-KG-Bench framework provides automatic evaluation and storage of LLM responses as well as statistical data and visualization tools to support tracking of prompt engineering and model performance.
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing Future Brain Education Technology Co., Ltd.（北京未来脑教育科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Guangxi Zhuang Autonomous Region Big Data Research Institute（广西壮族自治区大数据研究院）
Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking
Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II（那不勒斯费德里科二世大学电气工程与信息技术系）；Northwestern University, Department of Computer Science, McCormick School of Engineering and Applied Science（西北大学计算机科学系，麦科马克工程与应用科学学院）
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University（沃克生物医学工程部门，佐治亚理工学院和埃默里大学）；College of Future of Technology, Peking University（未来技术学院，北京大学）；School of Architecture, Tsinghua University（建筑学院，清华大学）；School of Electrical and Computer Engineering, Georgia Institute of Technology（电气与计算机工程学院，佐治亚理工学院）；School of Computer Science, Georgia Institute of Technology（计算机科学学院，佐治亚理工学院）
Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation
Department of Methodology and Statistics, Utrecht University, The Netherlands（方法论与统计学系，乌特列支大学，荷兰）；Department of Information and Computing Sciences, Utrecht University, The Netherlands（信息与计算科学系，乌特列支大学，荷兰）；Queen Mary University of London, London, United Kingdom（伦敦女王玛丽大学，伦敦，英国）
Generalization in medical AI: a perspective on developing scalable models
Faculty of Biomedical Engineering, Technion Israel Institute of Technology（以色列理工学院生物医学工程学院）；Institute for Medical Engineering and Science, Massachusetts Institute of Technology（麻省理工学院医学工程与科学研究所）；Harvard TH Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）
