微调大语言模型并评估检索方法以改进建筑规范问答
Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codes
AI总结：
本文研究基于建筑规范的问答系统，通过评估多种检索方法并微调语言模型，发现Elasticsearch检索效果最佳，且领域微调能提升模型生成能力，从而优化RAG系统在NBCC上的表现。
摘要：
建筑规范是为建筑设计、施工和安全制定标准的法规，旨在确保结构完整性、防火性和无障碍性。这些规范通常篇幅庞大、内容复杂且频繁更新，使得人工查询既困难又耗时。主要难点包括浏览大量文本、理解技术语言以及在不同章节中定位相关条款。一种潜在的解决方案是构建一个基于建筑规范回答用户查询的问答（QA）系统。在构建问答系统的各种方法中，检索增强生成（RAG）在性能上表现突出。RAG由两个组件组成：检索器和语言模型。本研究重点在于为建筑规范确定合适的检索器方法，并利用微调技术优化语言模型的生成能力。我们通过在加拿大国家建筑规范（NBCC）上执行检索，对各种检索方法进行了详细评估，并利用源自NBCC的数据集探索了领域特定微调对多个语言模型的影响。我们的分析包括对不同检索器的比较评估，以及预训练模型和微调模型的性能对比，以确定在NBCC数据集上进行微调对语言模型的有效性和领域特定适应性。实验结果表明，Elasticsearch被证明是所有检索器中最稳健的。研究结果还表明，在NBCC特定数据集上微调语言模型可以增强其生成上下文相关响应的能力。当与像Elasticsearch这样强大的检索器所检索到的上下文相结合时，这种大语言模型（LLM）性能的提升可以优化RAG系统，使其能够更好地应对NBCC的复杂性。
英文摘要：
Building codes are regulations that establish standards for the design, construction, and safety of buildings to ensure structural integrity, fire protection, and accessibility. They are often extensive, complex, and subject to frequent updates, making manual querying challenging and time-consuming. Key difficulties include navigating large volumes of text, interpreting technical language, and identifying relevant clauses across different sections. A potential solution is to build a Question-Answering (QA) system that answers user queries based on building codes. Among the various methods for building a QA system, Retrieval-Augmented Generation (RAG) stands out in performance. RAG consists of two components: a retriever and a language model. This study focuses on identifying a suitable retriever method for building codes and optimizing the generational capability of the language model using fine-tuning techniques. We conducted a detailed evaluation of various retrieval methods by performing the retrieval on the National Building Code of Canada (NBCC) and explored the impact of domain-specific fine-tuning on several language models using the dataset derived from NBCC. Our analysis included a comparative assessment of different retrievers and the performance of both pre-trained and fine-tuned models to determine the efficacy and domain-specific adaptation of language models using fine-tuning on the NBCC dataset. Experimental results showed that Elasticsearch proved to be the most robust retriever among all. The findings also indicate that fine-tuning language models on an NBCC-specific dataset can enhance their ability to generate contextually relevant responses. When combined with context retrieved by a powerful retriever like Elasticsearch, this improvement in LLM performance can optimize the RAG system, enabling it to better navigate the complexities of the NBCC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
