arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.16724cs.CL
收藏

面向大型语言模型的结构感知领域知识注入

Structure-aware Domain Knowledge Injection for Large Language Models

Zhejiang University(浙江大学);Alibaba Cloud(阿里云)

Kai Liu, Ze Chen, Zhihang Fu, Wei Zhang, Rongxin Jiang, Fan Zhou, Yaowu Chen, Yue Wu, Jieping Ye

AI总结:

提出 StructTuning 两阶段结构感知知识注入方法,仅用5%语料即可达到传统方法100%性能,并在多个基准上验证其优越性与可扩展性。

摘要:

本文介绍了一种名为 StructTuning 的开创性方法,用于高效地将基础大型语言模型(LLM)转化为领域专家。该方法将训练语料需求大幅降低至仅 5%,同时达到传统知识注入性能的 100%,表现令人瞩目。受结构化人类教育的启发,我们提出了一种新颖的两阶段知识注入与对齐策略:结构感知持续预训练(SCPT)和结构感知监督微调(SSFT)。在 SCPT 阶段,我们自动提取领域知识分类体系并重组训练语料,使 LLM 能够有效地将文本片段与分类体系中的目标知识点关联起来。在 SSFT 阶段,我们明确提示模型在输出中阐明潜在的知识结构,利用结构化的领域洞见解决实际问题。我们的最终方法在 LongBench 和 MMedBench 数据集上跨模型架构和规模进行了广泛评估,结果表明其性能优于其他知识注入方法。我们还探索了该方法在不同训练语料规模下的可扩展性,为以更好的数据利用增强领域专用 LLM 奠定了基础。

英文摘要:

This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly reduces the training corpus needs to a mere 5% while achieving an impressive 100% of traditional knowledge injection performance. Motivated by structured human education, we propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). In the SCPT phase, we automatically extract the domain knowledge taxonomy and reorganize the training corpora, enabling LLMs to effectively link textual segments to targeted knowledge points within the taxonomy. In the SSFT phase, we explicitly prompt models to elucidate the underlying knowledge structure in their outputs, leveraging the structured domain insight to address practical problems. Our ultimate method was extensively evaluated across model architectures and scales on LongBench and MMedBench datasets, demonstrating superior performance against other knowledge injection methods. We also explored our method's scalability across different training corpus sizes, laying the foundation to enhance domain-specific LLMs with better data utilization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑