利用支持长上下文的微调检索增强生成：面向3GPP标准
Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards
AI总结：
针对大语言模型处理电信技术标准的难题，提出基于Phi-2的微调RAG系统，结合语义分块、重排序、SelfExtend上下文扩展及LoRA微调，性能超GPT-4，为网络智能体语言模型奠基。
摘要：
近期研究表明，大语言模型（LLM）在处理电信领域技术标准时存在困难。我们提出了一种基于Phi-2小语言模型（SLM）的微调检索增强生成（RAG）系统，作为通信网络的预言机。所开发的系统采用前瞻性语义分块技术，根据嵌入相似度自适应确定解析断点，能够有效处理多种文档格式。为应对技术标准中存在多个相似上下文的挑战，我们采用重排序算法对检索到的最相关分块进行优先级排序。考虑到Phi-2上下文窗口较小的局限性，我们实现了SelfExtend这一最新技术，在推理阶段扩展上下文窗口，这不仅提升了性能，还能适配从普通用户到专业技术人员的更广泛用户查询与设计需求。微调阶段我们使用低秩适配（LoRA）技术提升训练计算效率，支持在小数据集上实现有效微调。综合实验表明，该系统相较电信领域现有问答方法取得了大幅提升，性能超过了GPT-4等规模约为其880倍的更大语言模型。本工作提出了一种将小语言模型应用于通信网络的新方法，实现了效率与性能的平衡，可作为面向网络的智能体语言模型的研究基础。
英文摘要：
Recent studies show that large language models (LLMs) struggle with technical standards in telecommunications. We propose a fine-tuned retrieval-augmented generation (RAG) system based on the Phi-2 small language model (SLM) to serve as an oracle for communication networks. Our developed system leverages forward-looking semantic chunking to adaptively determine parsing breakpoints based on embedding similarity, enabling effective processing of diverse document formats. To handle the challenge of multiple similar contexts in technical standards, we employ a re-ranking algorithm to prioritize the most relevant retrieved chunks. Recognizing the limitations of Phi-2's small context window, we implement a recent technique, namely SelfExtend, to expand the context window during inference, which not only boosts the performance but also can accommodate a wider range of user queries and design requirements from customers to specialized technicians. For fine-tuning, we utilize the low-rank adaptation (LoRA) technique to enhance computational efficiency during training and enable effective fine-tuning on small datasets. Our comprehensive experiments demonstrate substantial improvements over existing question-answering approaches in the telecom domain, achieving performance that exceeds larger language models such as GPT-4 (which is about 880 times larger in size). This work presents a novel approach to leveraging SLMs for communication networks, offering a balance of efficiency and performance. This work can serve as a foundation towards agentic language models for networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
