任务感知的反课程掩码提升文本下游性能
Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text
AI总结：
提出任务感知反课程掩码（TIACBM）方法，通过任务知识选择掩码标记和循环衰减掩码比例，在情感分析、主题分类和作者归属任务上显著提升性能。
摘要：
掩码语言建模已成为预训练大型语言模型（LLMs）的广泛采用的无人监督技术。然而，选择掩码标记的过程是随机的，且掩码标记的百分比在整个训练过程中通常是固定的。在本文中，我们提出基于一种新颖的任务感知反课程学习方案来调整掩码比例并决定掩码哪些标记。首先，我们利用关于有用和有害标记的任务特定知识来确定掩码哪些标记。其次，我们提出一种循环衰减的掩码比例，这对应于一种反课程安排（从难到易）。我们通过三种不同的下游任务来示例我们新颖的任务感知反课程掩码（TIACBM）方法：情感分析、按主题的文本分类和作者归属。我们的研究结果表明，TIACBM增强了模型聚焦于关键任务相关特征的能力，有助于在各项任务中实现统计显著的性能提升。我们在https://github.com/JarcaAndrei/TIACBM发布我们的代码。
英文摘要：
Masked language modeling has become a widely adopted unsupervised technique to pre-train large language models (LLMs). However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically fixed for the entire training process. In this paper, we propose to adjust the masking ratio and to decide which tokens to mask based on a novel task-informed anti-curriculum learning scheme. First, we harness task-specific knowledge about useful and harmful tokens in order to determine which tokens to mask. Second, we propose a cyclic decaying masking ratio, which corresponds to an anti-curriculum schedule (from hard to easy). We exemplify our novel task-informed anti-curriculum by masking (TIACBM) approach across three diverse downstream tasks: sentiment analysis, text classification by topic, and authorship attribution. Our findings suggest that TIACBM enhances the ability of the model to focus on key task-relevant features, contributing to statistically significant performance gains across tasks. We release our code at https://github.com/JarcaAndrei/TIACBM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
