解放Transformer模型：任务特定上下文归因有望在无需微调预训练LLM的情况下提升泛化能力
Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs
AI总结：
针对微调预训练LLM会损失泛化性的问题，本文提出基于任务特定概念算子的上下文归因框架，无需微调即可提升分类性能与泛化能力，在多个数据集上取得优于现有模型的效果且已开源。
摘要：
在特定数据集上微调大型预训练语言模型（LLM）是自然语言处理（NLP）分类任务中常用的策略。然而，这种方法通常会导致模型泛化能力的损失。本文提出了一个框架，该框架能够维持泛化能力，并通过利用任务特定上下文归因来提升下游任务的性能。我们表明，使用任务特定概念算子对来自任意Transformer模型的文本表示进行线性变换，会得到向潜在概念空间的投影，本文将其称为上下文归因。该特定概念算子在监督学习阶段通过新型损失函数进行优化。所提出的框架证明，针对每个任务目标对文本表示进行上下文归因能够提升判别函数的能力，从而在分类任务中取得更优的性能。在三个数据集（即HateXplain、IMDB评论和Social Media Attributions）上的实验结果表明，所提出的模型取得了更优的准确率和泛化能力。具体而言，对于HateXplain数据集上未经过微调的BERT，我们观察到准确率提升了8%，F1分数提升了10%。而对于IMDB数据集，该模型在准确率和F1分数上均比经过微调的最先进XLNet高出1%。此外，在域外交叉数据集测试中，在IMDB数据集上微调的DistilBERT与所提出的模型相结合，使HateXplain数据集上的F1分数提升了7%。对于由YouTube评论构成的Social Media Attributions数据集，我们观察到F1指标提升了5.2%。所提出的框架使用PyTorch实现，并在GitHub上开源提供。
英文摘要：
Fine-tuning large pre-trained language models (LLMs) on particular datasets is a commonly employed strategy in Natural Language Processing (NLP) classification tasks. However, this approach usually results in a loss of models generalizability. In this paper, we present a framework that allows for maintaining generalizability, and enhances the performance on the downstream task by utilizing task-specific context attribution. We show that a linear transformation of the text representation from any transformer model using the task-specific concept operator results in a projection onto the latent concept space, referred to as context attribution in this paper. The specific concept operator is optimized during the supervised learning stage via novel loss functions. The proposed framework demonstrates that context attribution of the text representation for each task objective can improve the capacity of the discriminator function and thus achieve better performance for the classification task. Experimental results on three datasets, namely HateXplain, IMDB reviews, and Social Media Attributions, illustrate that the proposed model attains superior accuracy and generalizability. Specifically, for the non-fine-tuned BERT on the HateXplain dataset, we observe 8% improvement in accuracy and 10% improvement in F1-score. Whereas for the IMDB dataset, fine-tuned state-of-the-art XLNet is outperformed by 1% for both accuracy and F1-score. Furthermore, in an out-of-domain cross-dataset test, DistilBERT fine-tuned on the IMDB dataset in conjunction with the proposed model improves the F1-score on the HateXplain dataset by 7%. For the Social Media Attributions dataset of YouTube comments, we observe 5.2% increase in F1-metric. The proposed framework is implemented with PyTorch and provided open-source on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
