Retrieval Augmented Generation (RAG) is widely used to enable Large Language Models (LLMs) perform Question Answering (QA) tasks in various domains. However, RAG based on open-source LLM for specialized domains has challenges of evaluating generated responses. A popular framework in the literature is the RAG Assessment (RAGAS), a publicly available library which uses LLMs for evaluation. One disadvantage of RAGAS is the lack of details of derivation of numerical value of the evaluation metrics. One of the outcomes of this work is a modified version of this package for few metrics (faithfulness, context relevance, answer relevance, answer correctness, answer similarity and factual correctness) through which we provide the intermediate outputs of the prompts by using any LLMs. Next, we analyse the expert evaluations of the output of the modified RAGAS package and observe the challenges of using it in the telecom domain. We also study the effect of the metrics under correct vs. wrong retrieval and observe that few of the metrics have higher values for correct retrieval. We also study for differences in metrics between base embeddings and those domain adapted via pre-training and fine-tuning. Finally, we comment on the suitability and challenges of using these metrics for in-the-wild telecom QA task.
First Token Probability Guided RAG for Telecom Question Answering
Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration
Emory University（埃默里大学）；University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；Georgia Institute of Technology（佐治亚理工学院）；SUNY Albany（纽约州立大学奥尔巴尼分校）
Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generation
School of Computer Science（计算机科学学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Standards and Mobility Innovation Lab（标准与移动创新实验室）；Samsung Research America（三星美国研究）
RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；Xiaohongshu Inc.（小红书公司）；Xi’an Jiaotong University（西安交通大学）
University of Macau（澳门大学）；Xiamen University（厦门大学）；Peking University（北京大学）；Hanyang University（翰阳大学）；University of Liverpool（利物浦大学）；Zhejiang University（浙江大学）
Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs
The University of Tokyo（东京大学）；NEC Laboratories America（NEC美国实验室）；Dokkyo Medical University（多摩医学大学）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）
Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoring
Faculty of Education, The University of Hong Kong（香港大学教育学院）；Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学土木与环境工程系）
NeuSym-RAG: Hybrid Neural Symbolic Retrieval with Multiview Structuring for PDF Question Answering
MoE Key Lab of Artificial Intelligence（摩尔电子关键实验室）；X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University（X-LANCE实验室，计算机科学学院，上海交通大学）；Jiangsu Key Lab of Language Computing（江苏语言计算重点实验室）；AISpeech Co., Ltd.（AISpeech公司）；Suzhou Laboratory（苏州实验室）
HyCE-RAG: Hypergraph Chain-of-Evidence Retrieval-Augmented Generation for Explainable Multi-hop Question Answering
University of Macau（澳门大学）；Shenzhen Kaihong Digital Industry Development Co., Ltd.（深圳开鸿数字产业发展有限公司）；University of New South Wales（新南威尔士大学）；Zhejiang University（浙江大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
RAG-BioQA: A Retrieval-Augmented Generation Framework for Long-Form Biomedical Question Answering
Department of CSE Emory University Georgia, USA（计算机科学与工程系 埃默里大学 美国佐治亚州）；Department of CSE SRM University AP Andhra Pradesh, India（计算机科学与工程系 SRM大学AP 安得拉邦印度）；Department of CSE Trine University Michigan, USA（计算机科学与工程系 特林大学 密歇根州美国）
