Large language models (LLMs) has become a significant research focus and is utilized in various fields, such as text generation and dialog systems. One of the most essential applications of LLM is Retrieval Augmented Generation (RAG), which greatly enhances generated content's reliability and relevance. However, evaluating RAG systems remains a challenging task. Traditional evaluation metrics struggle to effectively capture the key features of modern LLM-generated content that often exhibits high fluency and naturalness. Inspired by the RAGAS tool, a well-known RAG evaluation framework, we extended this framework into a KG-based evaluation paradigm, enabling multi-hop reasoning and semantic community clustering to derive more comprehensive scoring metrics. By incorporating these comprehensive evaluation criteria, we gain a deeper understanding of RAG systems and a more nuanced perspective on their performance. To validate the effectiveness of our approach, we compare its performance with RAGAS scores and construct a human-annotated subset to assess the correlation between human judgments and automated metrics. In addition, we conduct targeted experiments to demonstrate that our KG-based evaluation method is more sensitive to subtle semantic differences in generated outputs. Finally, we discuss the key challenges in evaluating RAG systems and highlight potential directions for future research.
Know3-RAG: A Knowledge-aware RAG Framework with Adaptive Retrieval, Generation, and Filtering
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China(认知智能国家重点实验室,安徽合肥,中国);The Hong Kong University of Science and Technology(香港科学与技术大学)
DO-RAG: A Domain-Specific QA Framework Using Knowledge Graph-Enhanced Retrieval-Augmented Generation
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Beijing National Research Center for Information Science and Technology - Tsinghua University(北京信息科学与技术国家研究中心-清华大学)
Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs
The University of Tokyo(东京大学);NEC Laboratories America(NEC美国实验室);Dokkyo Medical University(多摩医学大学);Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院)
RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integration
KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education?
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China(计算机科学与工程系,香港科技大学,香港特别行政区,中国);Department of Mechanical Engineering, The University of Tokyo, Tokyo, Japan(机械工程系,东京大学,东京,日本)
KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG
School of Biomedical Engineering, Tsinghua, China(清华大学生物医学工程学院);School of Informatics, Xiamen University, China(厦门大学信息学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学计算机科学与技术系);Dept. of Computer Science and Technology, Northeastern University, Shenyang, China(东北大学沈阳校区计算机科学与技术系);Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China(中国科学院信息工程研究所)
What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge
University of Oslo(奥斯陆大学);Bosch Center for AI(博世人工智能中心);University of Stuttgart(斯图加特大学);Amazon(亚马逊公司);University of Oxford(牛津大学);The University of Manchester(曼彻斯特大学);University of Southampton(南安普顿大学)
IA-RAG: Interval-Algebra-Driven Temporal Reasoning for Dynamic Knowledge Retrieval
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Shanghai for Science and Technology(上海科技大学);Harbin Engineering University(哈尔滨工程大学)
RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation
School of Computer Science and Technology,Xi'an Jiaotong University(西安交通大学计算机科学与技术学院);Department of Computer Science, The University of Manchester(曼彻斯特大学计算机科学系);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);National University of Singapore(新加坡国立大学)
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)