arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.03158cs.CLcs.AI
收藏

CoT驱动的短文本分类框架:从大模型到小模型的能力增强与迁移

CoT-Driven Framework for Short Text Classification: Enhancing and Transferring Capabilities from Large to Smaller Model

Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);University of Chinese Academy of Sciences(中国科学院大学);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院)

Hui Wu, Yuanben Zhang, Zhonghe Han, Yingyan Hou, Lei Wang, Siye Liu, Qihang Gong, Yunping Ge

AI总结:

提出SSE-CoT方法分解短文本分类任务以增强大语言模型能力,并设计CDMT框架将推理能力迁移至小模型,在六个基准上取得最优性能。

摘要:

短文本分类(STC)对于处理和理解当代数字平台上普遍存在的简短但内容丰富的文本至关重要。STC在把握语义和句法复杂性方面面临困难,这一问题在传统的预训练语言模型中表现明显。尽管图卷积网络通过整合外部知识库提升了性能,但这些方法受限于所用知识的质量和范围。近期,大语言模型(LLMs)和思维链(CoT)的出现显著提升了复杂推理任务的性能。然而,一些研究指出了它们在基础自然语言处理任务中应用的局限性。因此,本研究首先采用CoT来探究并增强LLMs在STC任务中的能力。我们提出了句法与语义增强思维链(SSE-CoT)方法,将STC任务有效分解为四个不同步骤:(i)关键概念识别,(ii)常识知识检索,(iii)文本重写,以及(iv)分类。此外,考虑到金融和医疗等领域的资源限制,我们随后引入了CoT驱动的多任务学习(CDMT)框架,将这些能力扩展到较小的模型。该框架首先从LLMs中提取推理依据,随后微调较小的模型以优化其性能。在六个短文本基准上的广泛实验验证了所提方法的有效性。特别是,SSE-CoT在所有数据集上均取得了显著改进的最先进性能,尤其在Ohsumed和TagMyNews数据集上表现突出。

英文摘要:

Short Text Classification (STC) is crucial for processing and understanding the brief but substantial content prevalent on contemporary digital platforms. The STC encounters difficulties in grasping the semantic and syntactic intricacies, an issue that is apparent in traditional pre-trained language models. Although Graph Convolutional Networks enhance performance by integrating external knowledge bases, these methods are limited by the quality and extent of the knowledge applied. Recently, the emergence of Large Language Models (LLMs) and Chain-of-Thought (CoT) has significantly improved the performance of complex reasoning tasks. However, some studies have highlighted the limitations of their application in fundamental NLP tasks. Consequently, this study first employs CoT to investigate and enhance the capabilities of LLMs in STC tasks. We propose the Syntactic and Semantic Enrichment CoT (SSE-CoT) method, effectively decomposing the STC tasks into four distinct steps: (i) essential concept identification, (ii) common-sense knowledge retrieval, (iii) text rewriting, and (iv) classification. Furthermore, recognizing resource constraints in sectors like finance and healthcare, we then introduce the CoT-Driven Multi-Task Learning (CDMT) framework to extend these capabilities to smaller models. This framework begins by extracting rationales from LLMs and subsequently fine-tunes smaller models to optimize their performance. Extensive experimentation across six short-text benchmarks validated the efficacy of the proposed methods. In particular, SSE-CoT achieved state-of-the-art performance with substantial improvements on all datasets, particularly on the Ohsumed and TagMyNews datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑