FourierKAN在文本分类头微调中优于MLP
FourierKAN outperforms MLP on Text Classification Head Fine-tuning
AI总结:
本研究提出Fourier KAN(FR-KAN)作为文本分类头,替代MLP,在七个预训练模型和四个任务上平均提升10%准确率和11% F1分数,且计算更高效、参数更少。
摘要:
在资源受限的环境中,对下游分类任务的适应涉及微调分类器的最后一层(即分类头),同时保持模型其余权重冻结。长期以来,使用预训练Transformer骨干网络微调的多层感知器(MLP)头一直是文本分类头微调的事实标准。然而,MLP的固定非线性往往难以完全捕捉预训练模型产生的上下文嵌入的细微差别,同时计算成本也较高。在我们的工作中,我们研究了KAN及其变体Fourier KAN(FR-KAN)作为替代文本分类头的有效性。我们的实验表明,FR-KAN显著优于MLP,在七个预训练Transformer模型和四个文本分类任务上,平均准确率提升10%,F1分数提升11%。除了性能提升外,FR-KAN计算效率更高,训练速度更快,参数更少。这些结果凸显了FR-KAN作为轻量级分类头的潜力,对推进其他自然语言处理(NLP)任务具有更广泛的意义。
英文摘要:
In resource constraint settings, adaptation to downstream classification tasks involves fine-tuning the final layer of a classifier (i.e. classification head) while keeping rest of the model weights frozen. Multi-Layer Perceptron (MLP) heads fine-tuned with pre-trained transformer backbones have long been the de facto standard for text classification head fine-tuning. However, the fixed non-linearity of MLPs often struggles to fully capture the nuances of contextual embeddings produced by pre-trained models, while also being computationally expensive. In our work, we investigate the efficacy of KAN and its variant, Fourier KAN (FR-KAN), as alternative text classification heads. Our experiments reveal that FR-KAN significantly outperforms MLPs with an average improvement of 10% in accuracy and 11% in F1-score across seven pre-trained transformer models and four text classification tasks. Beyond performance gains, FR-KAN is more computationally efficient and trains faster with fewer parameters. These results underscore the potential of FR-KAN to serve as a lightweight classification head, with broader implications for advancing other Natural Language Processing (NLP) tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。