DECSELFMASK: 通过自相关引导掩码利用未标记文本进行仅解码器分类
DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification
AI总结:
提出DecSelfMask方法,利用相关性归因引导掩码策略从无标签数据创建自监督训练样本,通过下一词预测重构掩码部分,提升仅解码器模型在分类任务上的性能,在136个临床任务上平均Macro F1提升19.9点。
摘要:
分类任务需要标注数据,但收集这些数据往往昂贵、耗时甚至不可行。医学领域尤其如此,大型数据集通常只有少量标注样本。为解决这一问题,我们提出DecSelfMask(通过掩码进行解码器自学习),一种增强仅解码器模型在分类任务上性能的方法。我们基于常见的自学习方法,利用模型从无标签数据创建训练样本,并提出一种新颖的相关性引导掩码策略。我们使用相关性归因方法确定未标注文本中与任务相关的部分。然后通过掩码这些部分创建自监督训练样本,训练模型通过下一词预测重建它们。我们假设这些样本传达了关于未标注数据结构和语义的知识,可能对下游性能有用。我们在来自一家意大利医院的190万份临床笔记的136个任务上测试了我们的方法。我们在5个不同规模和系列的模型上量化了DecSelfMask对下游任务的影响,包括探测分析。实验显示持续改进,优于标准监督微调方法(Macro F1提高19.9点)、合成标签生成(提高12.5点)和持续预训练(提高6.3点),以及常见基线。
英文摘要:
Classification tasks require annotated data, which can often be expensive, time-consuming, or even unfeasible to collect. This is the case of the medical domain, where large datasets often have few annotated examples. To address this, we propose DecSelfMask (Decoder Self-learning by Masking), an approach to enhance decoder-only performance on classification tasks. We build on common self-learning approaches by leveraging a model to create training examples from unlabeled data, and propose a novel relevance-guided masking strategy. We use relevance attribution methods to determine what portions of unannotated texts are relevant for a task. We then create self-supervised training examples by masking out those portions, training the model to reconstruct them via next-token-prediction. We hypothesize that those examples convey knowledge about the structure and semantics of unannotated data that can be useful for downstream performance. We test our approach on 136 tasks from a collection of 1.9M clinical notes from an Italian hospital. We quantify DecSelfMask's impact on downstream tasks on 5 models of different scales and families, including a probing analysis. Experiments show consistent gains, outperforming the base models (+9.1 points in Macro F1), continual pretraining (+6.3), synthetic label generation (+12.5), as well as common baselines. The results show that relevance attribution can serve as a powerful source of supervision for decoder learning from unlabeled data, beyond existing methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。