QMOS：利用问题掩码损失与选项打乱技术增强电信领域大语言模型
QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
AI总结：
本文提出QMOS方法，通过问题掩码损失和选项打乱技巧增强开源小模型在电信领域多项选择题问答中的表现，在RAG框架下将Falcon-7B和Phi-2的准确率分别提升至49.30%和84.65%。
摘要：
大语言模型（LLMs）在问答（QA）系统领域带来了重大进展。这些模型在解决多个学科中的复杂问题时表现出色。然而，由于领域特定词汇、复杂技术概念以及对精确回答的要求，将LLMs应用于电信等专业领域会带来额外障碍。近期工作已使用GPT-3.5在检索增强生成（RAG）框架下针对电信相关问题取得了显著准确率。尽管有这些进展，GPT-3.5等模型的实际应用仍受限于其专有性质和高计算需求。本文提出QMOS，一种创新方法，利用问题掩码（Question-Masked）损失和选项打乱（Option Shuffling）技巧来提升LLMs在电信领域回答多项选择题的性能。我们的重点是在增强的RAG框架内使用开源、规模较小的语言模型（Phi-2和Falcon-7B）。我们的多方面方法涉及对整个LLM-RAG流程的多项增强，包括微调、检索、提示工程和推理。我们的方法显著优于现有结果，使用Falcon-7B将准确率从基线24.70%提升至49.30%，使用Phi-2将准确率从42.07%提升至84.65%。
英文摘要：
Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
