Although automated metrics are commonly used to evaluate NLG systems, they often correlate poorly with human judgements. Newer metrics such as BERTScore have addressed many weaknesses in prior metrics such as BLEU and ROUGE, which rely on n-gram matching. These newer methods, however, are still limited in that they do not consider the generation context, so they cannot properly reward generated text that is correct but deviates from the given reference.
In this paper, we propose Language Model Augmented Relevance Score (MARS), a new context-aware metric for NLG evaluation. MARS leverages off-the-shelf language models, guided by reinforcement learning, to create augmented references that consider both the generation context and available human references, which are then used as additional references to score generated text. Compared with seven existing metrics in three common NLG tasks, MARS not only achieves higher correlation with human reference judgements, but also differentiates well-formed candidates from adversarial samples to a larger degree.
LRP4RAG: Detecting Hallucinations in Retrieval-Augmented Generation via Layer-wise Relevance Propagation
Alibaba Cloud（阿里云）；School of Computer Sciences, Universiti Sains Malaysia（计算机科学学院，马来西亚国立大学）；Department of Service Science and Operations Management, School of Management, Zhejiang University（服务科学与运营管理系，浙江大学）；State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）
REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；National University of Singapore（新加坡国立大学）；Baidu Inc.（百度公司）；School of Information, Renmin University of China（中国人民大学信息学院）
Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation
National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；City University of Hong Kong（香港城市大学）；Tencent Technology（腾讯科技）；Peking University（北京大学）
DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；Swiss Finance Institute (SFI)（瑞士金融研究所）；University of Electronic Science and Technology of China（电子科技大学）
RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China（语音与语言信息处理国家工程研究中心，中国科学技术大学）；State Key Laboratory of Cognitive Intelligence, iFLYTEK Research（认知智能国家重点实验室，iFLYTEK研究院）
Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Baidu Inc（百度公司）
Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models
Universitas Islam Riau（廖内伊斯兰大学）；Universitas Lancang Kuning（兰昌坤宁大学）；Izmir Katip Celebi University（伊兹密尔卡蒂普·切莱比大学）；Ritsumeikan University（立命馆大学）；Universiti Islam Antarabangsa Tuanku Syed Sirajuddin (UniSIRAJ)（端姑赛德·西拉鲁丁国际伊斯兰大学（UniSIRAJ））；International Islamic University Malaysia（马来西亚国际伊斯兰大学）
SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context
School of Computer Science, University of Science and Technology of China（中国科学技术大学计算机科学学院）；School of Biomedical Engineering, USTC（USTC生物医学工程学院）；Suzhou Institute for Advanced Reasearch, USTC（USTC苏州市先进研究所）
A Large Language Model-based Framework for Semi-Structured Tender Document Retrieval-Augmented Generation
School of Information Management Sun Yat-sen University Guangzhou, China；School of Information Management Sun Yat-sen University Guangzhou, China corresponding author
CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation
School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；School of Information Engineering, Dalian Ocean University（大连海洋大学信息工程学院）；School of Information Engineering, Liaodong University（辽东学院信息工程学院）；Information Technology Center, Qinghai University（青海大学信息技术中心）；College of Health-Preservation and Wellness, Dalian Medical University（大连医科大学养生与健康学院）；Tencent（腾讯）
A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering
Department of Computer Science, Chungbuk National University（Chungbuk National University 计算机科学系）；Department of Big Data, Chungbuk National University（Chungbuk National University 大数据系）；General Department of Information and Communication Technology, Ministry of Post and Telecommunications（邮电部信息和通信技术总局）；Department of Management Information Systems, Chungbuk National University（Chungbuk National University 管理信息系统系）；BigDatalabs Co., Ltd（BigDatalabs 公司）
Self-Calibrated Listwise Reranking with Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；NExT++ Research Centre, National University of Singapore（新加坡国立大学NExT++研究中心）；Baidu Inc.（百度公司）
