大型语言模型(LLMs)展现出卓越的能力,但往往由于仅依赖其内部嵌入的知识而产生不准确的响应。检索增强生成(RAG)通过引入外部信息检索系统来增强LLMs,在查询之外提供额外的上下文,以减轻特定上下文下的不准确问题。然而,准确性问题依然存在,因为模型可能依赖不相关的文档,或从其训练知识中进行错误的推断。为了评估和改进RAG框架中检索系统和LLM的性能,我们提出了VERA(Validation and Enhancement for Retrieval Augmented systems,检索增强系统的验证与增强),该系统旨在:1)在生成响应之前评估并增强检索到的上下文;2)评估并优化LLM生成的响应,以确保精确性并最小化错误。VERA采用了一个兼具评估与增强功能的LLM,它首先检查是否需要外部检索,评估检索上下文的相关性和冗余度,并对其进行优化以消除非必要信息。在响应生成之后,VERA将响应拆分为原子陈述,评估它们与查询的相关性,并确保其遵循上下文。我们的实验表明,VERA不仅在提升较小的开源模型性能方面表现出显著效果,对大型最先进模型同样如此。这些增强突显了VERA生成准确且相关响应的潜力,推动了检索增强语言建模领域的最新进展。VERA结合了多个评估和优化步骤的稳健方法,有效缓解了幻觉现象并改进了检索和响应过程,使其成为在信息生成中要求高准确性和可靠性的应用的宝贵工具。
英文摘要:
Large language models (LLMs) exhibit remarkable capabilities but often produce inaccurate responses, as they rely solely on their embedded knowledge. Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating an external information retrieval system, supplying additional context along with the query to mitigate inaccuracies for a particular context. However, accuracy issues still remain, as the model may rely on irrelevant documents or extrapolate incorrectly from its training knowledge. To assess and improve the performance of both the retrieval system and the LLM in a RAG framework, we propose \textbf{VERA} (\textbf{V}alidation and \textbf{E}nhancement for \textbf{R}etrieval \textbf{A}ugmented systems), a system designed to: 1) Evaluate and enhance the retrieved context before response generation, and 2) Evaluate and refine the LLM-generated response to ensure precision and minimize errors. VERA employs an evaluator-cum-enhancer LLM that first checks if external retrieval is necessary, evaluates the relevance and redundancy of the retrieved context, and refines it to eliminate non-essential information. Post-response generation, VERA splits the response into atomic statements, assesses their relevance to the query, and ensures adherence to the context. Our experiments demonstrate VERA's remarkable efficacy not only in improving the performance of smaller open-source models, but also larger state-of-the art models. These enhancements underscore VERA's potential to produce accurate and relevant responses, advancing the state-of-the-art in retrieval-augmented language modeling. VERA's robust methodology, combining multiple evaluation and refinement steps, effectively mitigates hallucinations and improves retrieval and response processes, making it a valuable tool for applications demanding high accuracy and reliability in information generation. .
A Graph-Enhanced Defense Framework for Explainable Fake News Detection with LLM
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Hong Kong Baptist University(香港浸会大学);Guangdong Institute of Smart Education, Jinan University(暨南大学广东智慧教育研究院);University of Science and Technology Beijing(北京科技大学);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence(知识驱动人机智能教育部工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)
Hybrid Retrieval-Augmented Generation Agent for Trustworthy Legal Question Answering in Judicial Forensics
Department of Electronic Engineering, City University of Hong Kong(DongGuan)(香港城市大学(东莞)电子工程系);Department of Electronic Engineering, City University of Hong Kong(香港城市大学电子工程系)
A Question Answering Dataset for Temporal-Sensitive Retrieval-Augmented Generation
Laboratory for Big Data and Decision, National University of Defense Technology, Changsha, China(大数据与决策实验室,国防科技大学,长沙,中国);Baidu Inc., Beijing, China(百度公司,北京,中国);Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China(计算机科学与技术系,哈尔滨工业大学(深圳),深圳,中国)
Large Language Models (LLMs) as Agents for Augmented Democracy
University of Toulouse(图卢兹大学);Corvinus University of Budapest(布达佩斯考文纽斯大学);IRIT, Université Toulouse Capitole(图卢兹 Capitole 大学图卢兹计算机研究所);Toulouse School of Economics(图卢兹经济学院);Université Toulouse Capitole(图卢兹 Capitole 大学)
Enhancing textual textbook question answering with large language models and retrieval augmented generation
Faculty of Computing and Information Technology, King Abdulaziz University(阿卜杜勒阿齐兹国王大学计算与信息技术学院);College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University (IMSIU)(伊玛目穆罕默德·本·沙特伊斯兰大学计算机与信息科学学院);School of Computing, Macquarie University(麦考瑞大学计算学院)
Augmenting Research Ideation with Data: An Empirical Investigation in Social Science
Wangxuan Institute of Computer Technology(王轩计算机技术研究所);Yuanpei College(元培学院);School of Government(政府学院);School of International Studies(国际研究学院);Institute for Carbon Neutrality(碳中和研究所);Analytics Lab for Global Risk Politics(全球风险政治分析实验室)
Automated Construction of Medical Indicator Knowledge Graphs Using Retrieval Augmented Large Language Models
The second hospital of Jilin University Northeast Asia Active Aging Laboratory Jilin, China Email;QMUL DERI \ Institute Of Health Informatics, Lonodn, UK Email;QMUL School of Electronic Engineering \ Computer Science London, UK Email;The second hospital of Jilin University College of Artificial Intelligence, \ University, Jilin, China Email
Human Mobility Modeling with Household Coordination Activities under Limited Information via Retrieval-Augmented LLMs
UCLA Mobility Lab, Department of Civil and Environmental Engineering, University of California, Los Angeles, Los Angeles, USA(加州大学洛杉矶分校移动实验室,土木与环境工程系,加州大学洛杉矶分校,洛杉矶,美国);Civil and Environmental Engineering Department, University of Louisville, Louisville, Kentucky, USA(路易斯维尔大学土木与环境工程系,路易斯维尔,肯塔基州,美国);Novateur Research Solutions, Ashburn, VA, USA(Novateur研究解决方案,弗吉尼亚州阿什本,美国)
a1: Steep Test-time Scaling Law via Environment Augmented Generation
AI Safety of Chinese Academy of Sciences, Institute of Computing Technology, CAS(中国科学院人工智能安全研究院计算技术研究所);University of California, Merced(加州大学默塞德分校);UBS AG(瑞银集团);University of Chinese Academy of Sciences(中国科学院大学)