Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);National Science Library, Chinese Academy of Sciences(中国科学院国家科学图书馆)
Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, Jun Zhao
In this paper, we propose $\textbf{Ne}$ural-$\textbf{Sy}$mbolic $\textbf{C}$ollaborative $\textbf{D}$istillation ($\textbf{NesyCD}$), a novel knowledge distillation method for learning the complex reasoning abilities of Large Language Models (LLMs, e.g., \textgreater 13B). We argue that complex reasoning tasks are difficult for Small Language Models (SLMs, e.g., $\leq$ 7B), as these tasks demand not only general cognitive abilities but also specialized knowledge, which is often sparse and difficult for these neural-based SLMs to effectively capture. Therefore, NesyCD distills the general capabilities and specialized knowledge in LLMs using different manners. On the one hand, we distill only general abilities from teacher LLMs into the student SLMs of parameterized neural networks. On the other hand, for the specialized abilities and uncommon knowledge of a complex reasoning task, we employ a symbolic knowledge distillation approach to obtain and store the specialized knowledge within a symbolic knowledge base (KB). By decoupling general and specialized capabilities, the proposed NesyCD can achieve superior performance cost-effectively, utilizing smaller models and blending parameterized neural networks with symbolic KB. Moreover, the specialized KB generalizes well and is comprehended and manipulated by humans. Our experiments show that NesyCD significantly boosts SLMs' complex reasoning performance on in-domain (BBH, GSM8K) and out-of-domain (AGIEval, ARC) datasets. Notably, our approach enabled the LLaMA3-8B and Qwen2-7B to surpass GPT-3.5-turbo in performance and come close to matching LLaMA3-70B, despite the latter having nine times more parameters. Our code will be available at https://github.com/Xnhyacinth/NesyCD.
Advancing Multi-Party Dialogue Framework with Speaker-ware Contrastive Learning
School of Computer Science and Engineering, Northwestern Polytechnical University(计算机科学与工程学院,西北工业大学);School of Computer Science and Technology, Xidian University(计算机科学与技术学院,西安电子科技大学);School of Artificial Intelligence and Big Data, Henan University of Technology(人工智能与大数据学院,河南理工大学)
Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysis
Department of Computer Science, National University of Singapore(计算机科学系,国立新加坡大学);Amazon Research, Amazon.com Inc(亚马逊研究,亚马逊公司);Department of Social Work, National University of Singapore(社会工作系,国立新加坡大学)
The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP
University of California, Los Angeles(加州大学洛杉矶分校);Georgia Institute of Technology(佐治亚理工学院);University of Wisconsin - Madison(威斯康星大学麦迪逊分校);University of Cape Coast(科克伍德大学);Carleton University(卡尔顿大学);Stetson University(史蒂文森大学);Northwestern University in Qatar(卡塔尔西北大学);Cornell University(康奈尔大学);Soka University of America(美国立命大学);Columbia University(哥伦比亚大学)
Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国);Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Hong Kong University of Science and Technology, Hong Kong, China(香港科技大学,香港,中国);Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团)