arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.18344cs.CLcs.AIcs.LG
收藏

聚合知识模型:利用微调和检索增强生成模型增强特定领域问答

Aggregated Knowledge Model: Enhancing Domain-Specific QA with Fine-Tuned and Retrieval-Augmented Generation Models

Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of California, Berkeley(加州大学伯克利分校);Convergent Computing/CCO

Fengchen Liu, Jordan Jung, Wei Feinstein, Jeff DAmbrogia, Gary Jung

AI总结:

面向 LBL ScienceIT 封闭领域问答,本文比较两类微调 LLM 与五个 RAG 模型,并提出用 K-means 聚合七模型答案的 AKM,实验表明其性能显著提升。

摘要:

本文介绍了一种增强封闭领域问答(QA)系统的新方法,重点关注劳伦斯伯克利国家实验室(LBL)科学信息技术(ScienceIT)领域的特定需求。本研究利用源自 ScienceIT 文档的丰富数据集,对两个微调的大型语言模型和五个检索增强生成(RAG)模型进行了详细比较。通过数据处理技术,我们将文档转换为结构化的上下文-问题-答案三元组,并利用最新的大型语言模型(AWS Bedrock、GCP PaLM2、Meta LLaMA2、OpenAI GPT-4、Google Gemini-Pro)获得数据驱动的洞察。此外,我们提出了聚合知识模型(AKM),该模型使用 K-means 聚类从上述七个模型中综合响应,以选择最具代表性的答案。跨多个指标对这些模型进行的评估全面考察了它们在 LBL ScienceIT 环境中的有效性和适用性。结果证明了整合微调和检索增强策略的潜在收益,突出了 AKM 所实现的显著性能提升。从本研究中获得的洞察可用于开发针对特定领域定制的专业化问答系统。

英文摘要:

This paper introduces a novel approach to enhancing closed-domain Question Answering (QA) systems, focusing on the specific needs of the Lawrence Berkeley National Laboratory (LBL) Science Information Technology (ScienceIT) domain. Utilizing a rich dataset derived from the ScienceIT documentation, our study embarks on a detailed comparison of two fine-tuned large language models and five retrieval-augmented generation (RAG) models. Through data processing techniques, we transform the documentation into structured context-question-answer triples, leveraging the latest Large Language Models (AWS Bedrock, GCP PaLM2, Meta LLaMA2, OpenAI GPT-4, Google Gemini-Pro) for data-driven insights. Additionally, we introduce the Aggregated Knowledge Model (AKM), which synthesizes responses from the seven models mentioned above using K-means clustering to select the most representative answers. The evaluation of these models across multiple metrics offers a comprehensive look into their effectiveness and suitability for the LBL ScienceIT environment. The results demonstrate the potential benefits of integrating fine-tuning and retrieval-augmented strategies, highlighting significant performance improvements achieved with the AKM. The insights gained from this study can be applied to develop specialized QA systems tailored to specific domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑