检索增强生成(Retrieval-Augmented Generation,RAG)系统在各行各业中被广泛用于查询闭域和内部知识库。然而,由于闭域数据的私有性以及缺乏具有可验证真实答案的查询,评估这些系统面临重大挑战。此外,目前缺乏分析方法来诊断存在问题的模块并识别失败类型,例如由知识缺陷或鲁棒性问题引起的失败。为应对这些挑战,我们提出GRAMMAR(GRounded And Modular Methodology for Assessment of RAG,即基于事实且模块化的RAG评估方法论),这是一个评估框架,包含一个基于事实的数据生成过程和一个能够有效定位缺陷模块的评估协议。我们的验证实验表明,GRAMMAR为识别脆弱模块提供了一种可靠的方法,并支持对文本形式脆弱性进行假设检验。该框架附带的开源工具可在我们的GitHub仓库中获取(见 https://github.com/xinzhel/grammar),便于轻松复现我们的结果,并支持在闭域环境中进行可靠且模块化的评估。
英文摘要:
Retrieval-Augmented Generation (RAG) systems are widely used across various industries for querying closed-domain and in-house knowledge bases. However, evaluating these systems presents significant challenges due to the private nature of closed-domain data and a scarcity of queries with verifiable ground truths. Moreover, there is a lack of analytical methods to diagnose problematic modules and identify types of failure, such as those caused by knowledge deficits or issues with robustness. To address these challenges, we introduce GRAMMAR (GRounded And Modular Methodology for Assessment of RAG), an evaluation framework comprising a grounded data generation process and an evaluation protocol that effectively pinpoints defective modules. Our validation experiments reveal that GRAMMAR provides a reliable approach for identifying vulnerable modules and supports hypothesis testing for textual form vulnerabilities. An open-source tool accompanying this framework is available in our GitHub repository (see https://github.com/xinzhel/grammar), allowing for easy reproduction of our results and enabling reliable and modular evaluation in closed-domain settings.
Agentic AI with Orchestrator-Agent Trust: A Modular Visual Classification Framework with Trust-Aware Orchestration and RAG-Based Reasoning
University of the Peloponnese, Department of Informatics and Telecommunications(希腊皮埃蒙特大学信息与电信系);Cornell University, Department of Biological and Environmental Engineering(康奈尔大学生物与环境工程系)
PlantMarkerBench: A Multi-Species Benchmark for Evidence-Grounded Plant Marker Reasoning
Department of Computer Science, Virginia Tech(弗吉尼亚理工学院计算机科学系);School of Plant and Environmental Sciences, Virginia Tech(弗吉尼亚理工学院植物与环境科学学院);Fralin Biomedical Research Institute, Virginia Tech(弗吉尼亚理工学院弗拉林生物医学研究学院);FBRI Cancer Research Center, Washington, DC(华盛顿特区FBRI癌症研究中心)
ChainReaction: Causal Chain-Guided Reasoning for Modular and Explainable Causal-Why Video Question Answering
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore(新加坡高性能计算研究所,科技研究局);Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore(科技研究局前沿人工智能研究中心,新加坡);College of Computing and Data Science, Nanyang Technological University, Singapore(新加坡南洋理工大学计算与数据科学学院)
Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis
University of Ljubljana, Faculty of Computer and Information Science(卢布尔雅那大学计算机与信息科学学院);University of Ljubljana, Faculty of Arts(卢布尔雅那大学艺术学院);Jožef Stefan Institute(乔塞夫·斯塔芬研究所)
Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models
MaiNLP, Center for Information and Language Processing, LMU Munich(MaiNLP、信息与语言处理中心、慕尼黑大学);Munich Center for Machine Learning (MCML), Munich(慕尼黑机器学习中心);Language Technology Group, University of Oslo(语言技术组、奥斯陆大学)
Decomposed Opinion Summarization with Verified Aspect-Aware Modules
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);School of Informatics, The University of Edinburgh(爱丁堡大学信息学院)
ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents
The Chinese University of Hong Kong, Hong Kong SAR, China(香港中文大学);Hong Kong Polytechnic University, Hong Kong SAR, China(香港理工大学);National University of Singapore, Singapore, Singapore(新加坡国立大学);Huawei Technologies, Hong Kong SAR, China(华为技术有限公司);Independent(独立)
Large Language Model Agent for Modular Task Execution in Drug Discovery
Department of Chemical Engineering, Carnegie Mellon University(化学工程系,卡内基梅隆大学);Department of Material Science and Engineering, Carnegie Mellon University(材料科学与工程系,卡内基梅隆大学);Department of Mechanical Engineering, Carnegie Mellon University(机械工程系,卡内基梅隆大学)