The integration of Large Language Models (LLMs) with Knowledge Graphs (KGs) offers significant synergistic potential for knowledge-driven applications. One possible integration is the interpretation and generation of formal languages, such as those used in the Semantic Web, with SPARQL being a core technology for accessing KGs. In this paper, we focus on measuring out-of-the box capabilities of LLMs to work with SPARQL and more specifically with SPARQL SELECT queries applying a quantitative approach.
We implemented various benchmarking tasks in the LLM-KG-Bench framework for automated execution and evaluation with several LLMs. The tasks assess capabilities along the dimensions of syntax, semantic read, semantic create, and the role of knowledge graph prompt inclusion.
With this new benchmarking tasks, we evaluated a selection of GPT, Gemini, and Claude models. Our findings indicate that working with SPARQL SELECT queries is still challenging for LLMs and heavily depends on the specific LLM as well as the complexity of the task. While fixing basic syntax errors seems to pose no problems for the best of the current LLMs evaluated, creating semantically correct SPARQL SELECT queries is difficult in several cases.
SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection
img/wse-cube-no-shadow.png Web \& Software Engineering (WSE) Research Group, Leipzig University of Applied Sciences (HTWK Leipzig), Leipzig, Germany CBZ München GmbH, Heilbronn, Germany
GGC: Selective Query Correction for Reliable Text-to-SPARQL Generation
Nanyang Technological University(南洋理工大学);Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所)
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Beijing Future Brain Education Technology Co., Ltd.(北京未来脑教育科技有限公司);The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);Guangxi Zhuang Autonomous Region Big Data Research Institute(广西壮族自治区大数据研究院)
NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalism
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Institute for Advanced Algorithms Research, China(中国高等算法研究院);Northeastern University, China(中国东北大学);State Key Laboratory of Media Convergence Production Technology and Systems, China(中国媒体融合生产技术与系统国家重点实验室)
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Tencent Robotics X(腾讯Robotics X实验室);University of Toronto(多伦多大学);Zhili College, Tsinghua University(清华大学志道书院)
Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain
Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification
University of Warwick(沃里克大学);Queen Mary University of London(伦敦大学女王学院);University of Bologna(博洛尼亚大学);King’s College London(伦敦大学国王学院);The Alan Turing Institute(艾伦·图灵研究所)
LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
Beijing University of Posts and Telecommunications(北京邮电大学);Paradigm Inc.(第四范式公司);Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University(沃克生物医学工程部门,佐治亚理工学院和埃默里大学);College of Future of Technology, Peking University(未来技术学院,北京大学);School of Architecture, Tsinghua University(建筑学院,清华大学);School of Electrical and Computer Engineering, Georgia Institute of Technology(电气与计算机工程学院,佐治亚理工学院);School of Computer Science, Georgia Institute of Technology(计算机科学学院,佐治亚理工学院)
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
GeoDS Lab, Department of Geography, University of Wisconsin-Madison, USA(威斯康星大学麦迪逊分校地理系GeoDS实验室);Department of Geography and Regional Research, University of Vienna, Austria(维也纳大学地理与区域研究系);Department of Geography, University of California-Santa Barbara, USA(加州大学圣塔芭芭拉分校地理系)
Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学);Sun Yat-sen University(中山大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院)
StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Beijing University of Posts and Telecommunications(北京邮电大学)
LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues
Faculty of Information Technology, Monash University, Australia(墨尔本大学信息技术学院,澳大利亚);School of Information Technology, Monash University Malaysia(墨尔本大学马来西亚分校信息学院);School of Business, Monash University Malaysia(墨尔本大学马来西亚分校商学院);Faculty of Law, Monash University, Australia(墨尔本大学法学院,澳大利亚)