arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.13203cs.CL
收藏

神经符号协作蒸馏:推进小型语言模型处理复杂推理任务

Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);National Science Library, Chinese Academy of Sciences(中国科学院国家科学图书馆)

Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, Jun Zhao

AI总结:

提出NesyCD神经符号协作蒸馏框架,将LLM通用能力蒸馏到SLM,并用符号KB承载专门知识,使7B至8B模型在复杂推理上接近70B模型。

摘要:

本文提出 Neural-Symbolic Collaborative Distillation(NesyCD,神经符号协作蒸馏),这是一种新颖的知识蒸馏方法,用于学习大型语言模型(LLM,例如参数规模大于13B的模型)的复杂推理能力。我们认为,复杂推理任务对小型语言模型(SLM,例如参数规模不超过7B的模型)而言十分困难,因为这些任务不仅需要通用认知能力,还需要专门知识,而这类知识通常较为稀疏,基于神经的SLM难以有效捕捉。因此,NesyCD以不同方式蒸馏LLM中的通用能力和专门知识。一方面,我们仅将教师LLM的通用能力蒸馏到参数化神经网络的学生SLM中。另一方面,对于复杂推理任务的专门能力和非通用知识,我们采用符号知识蒸馏方法,将专门知识获取并存储在符号知识库(KB)中。通过解耦通用能力与专门能力,所提出的NesyCD能够利用更小的模型,并将参数化神经网络与符号KB相结合,以高成本效益实现优越性能。此外,专门KB具有良好的泛化性,并且能够被人类理解和操作。实验表明,NesyCD显著提升了SLM在域内数据集(BBH、GSM8K)和域外数据集(AGIEval、ARC)上的复杂推理性能。值得注意的是,我们的方法使LLaMA3-8B和Qwen2-7B在性能上超过GPT-3.5-turbo,并接近LLaMA3-70B,尽管后者的参数量是前者的九倍。代码将在https://github.com/Xnhyacinth/NesyCD提供。

英文摘要:

In this paper, we propose $\textbf{Ne}$ural-$\textbf{Sy}$mbolic $\textbf{C}$ollaborative $\textbf{D}$istillation ($\textbf{NesyCD}$), a novel knowledge distillation method for learning the complex reasoning abilities of Large Language Models (LLMs, e.g., \textgreater 13B). We argue that complex reasoning tasks are difficult for Small Language Models (SLMs, e.g., $\leq$ 7B), as these tasks demand not only general cognitive abilities but also specialized knowledge, which is often sparse and difficult for these neural-based SLMs to effectively capture. Therefore, NesyCD distills the general capabilities and specialized knowledge in LLMs using different manners. On the one hand, we distill only general abilities from teacher LLMs into the student SLMs of parameterized neural networks. On the other hand, for the specialized abilities and uncommon knowledge of a complex reasoning task, we employ a symbolic knowledge distillation approach to obtain and store the specialized knowledge within a symbolic knowledge base (KB). By decoupling general and specialized capabilities, the proposed NesyCD can achieve superior performance cost-effectively, utilizing smaller models and blending parameterized neural networks with symbolic KB. Moreover, the specialized KB generalizes well and is comprehended and manipulated by humans. Our experiments show that NesyCD significantly boosts SLMs' complex reasoning performance on in-domain (BBH, GSM8K) and out-of-domain (AGIEval, ARC) datasets. Notably, our approach enabled the LLaMA3-8B and Qwen2-7B to surpass GPT-3.5-turbo in performance and come close to matching LLaMA3-70B, despite the latter having nine times more parameters. Our code will be available at https://github.com/Xnhyacinth/NesyCD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑