《古兰经研究中检索增强生成的探究：针对13个开源大语言模型的研究》
Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models
AI总结：
本研究针对古兰经研究领域的问答任务，引入RAG技术结合古兰经章描述数据集，评估13个不同规模开源大语言模型的表现，发现大模型整体更优，优化后的小模型也有亮眼表现，并探讨了模型规模、效率与回答质量的权衡。
摘要：
在将大语言模型（LLM）应用于《古兰经》研究相关问答这类敏感且特定领域的任务时，生成准确且符合上下文原意的回答至关重要。通用大语言模型常存在幻觉问题，生成的回答会偏离权威来源，引发了人们对其在宗教场景中可靠性的担忧。这一挑战凸显了开发既能整合特定领域知识，又能保证回答准确性、相关性和忠实度的系统的必要性。本研究探究了13个开源大语言模型，按规模分为大模型（如Llama3:70b、Gemma2:27b、QwQ:32b）、中模型（如Gemma2:9b、Llama3:8b）和小模型（如Llama3.2:3b、Phi3:3.8b）三类。研究采用检索增强生成（Retrieval-Augmented Generation，RAG）技术，以弥补单独使用各模型存在的问题。本研究使用了包含114个《古兰经》章的含义、历史背景和特点的描述性数据集，让模型在生成回答前先获取相关知识。研究通过人类评估者设定的三项核心指标对模型进行评估：上下文相关性、回答忠实度和回答相关性。研究结果显示，大模型在捕捉查询语义、生成准确且基于上下文的回答方面，表现始终优于小模型。Llama3.2:3b模型虽属于小模型，但在忠实度（得分4.619）和相关性（得分4.857）上表现优异，证明了经过充分优化的小型架构具备良好潜力。本文探讨了在特定领域应用中使用大语言模型时，模型规模、计算效率与回答质量之间的权衡关系。
英文摘要：
Accurate and contextually faithful responses are critical when applying large language models (LLMs) to sensitive and domain-specific tasks, such as answering queries related to quranic studies. General-purpose LLMs often struggle with hallucinations, where generated responses deviate from authoritative sources, raising concerns about their reliability in religious contexts. This challenge highlights the need for systems that can integrate domain-specific knowledge while maintaining response accuracy, relevance, and faithfulness. In this study, we investigate 13 open-source LLMs categorized into large (e.g., Llama3:70b, Gemma2:27b, QwQ:32b), medium (e.g., Gemma2:9b, Llama3:8b), and small (e.g., Llama3.2:3b, Phi3:3.8b). A Retrieval-Augmented Generation (RAG) is used to make up for the problems that come with using separate models. This research utilizes a descriptive dataset of Quranic surahs including the meanings, historical context, and qualities of the 114 surahs, allowing the model to gather relevant knowledge before responding. The models are evaluated using three key metrics set by human evaluators: context relevance, answer faithfulness, and answer relevance. The findings reveal that large models consistently outperform smaller models in capturing query semantics and producing accurate, contextually grounded responses. The Llama3.2:3b model, even though it is considered small, does very well on faithfulness (4.619) and relevance (4.857), showing the promise of smaller architectures that have been well optimized. This article examines the trade-offs between model size, computational efficiency, and response quality while using LLMs in domain-specific applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
