解锁效率：面向基因Transformer模型的自适应掩码策略
Unlocking Efficiency: Adaptive Masking for Gene Transformer Models
AI总结：
本文提出基于点互信息难度准则的课程掩码基因掩码策略CM-GEMS，通过逐步增加掩码预测难度提升基因Transformer预训练效率，在少样本和全数据集基因分类任务中以远少于基线模型的训练步数达到或接近最先进性能。
摘要：
基因Transformer模型（如Nucleotide Transformer、DNABert和LOGO）通过在完整人类参考基因组上使用掩码语言建模（Masked Language Modeling, MLM）训练目标来学习最优基因序列表示。然而，典型的标记化方法采用基本的滑动窗口标记（如k-mers），未能利用以基因为中心的语义。这可能导致对易于预测的序列进行（平凡的）掩码，从而造成MLM训练效率低下。已知时变训练策略能够提升语言和视觉任务中的预训练效率。在本工作中，我们专注于使用课程掩码（curriculum masking），通过基于点互信息（Pointwise Mutual Information）的难度准则系统地增加掩码标记预测任务的难度，因为基因序列缺乏类似于NLP领域中词或句子的明确定义语义单元。我们提出的基于课程掩码的基因掩码策略（Curriculum Masking-based Gene Masking Strategy, CM-GEMS）在下游基因序列分类任务中评估时，相比基线掩码方法展现出更优的表示学习能力。我们在少样本（五个数据集）和全数据集设置（包含27个任务的Genomic Understanding Evaluation基准）中进行了广泛评估。我们的研究结果表明，CM-GEMS优于在120K步训练的最先进模型（DNABert-2、Nucleotide Transformer、DNABert），仅用10K和1K步即可达到相似结果。我们还证明，课程学习的LOGO（一个2层DNABert类模型）能够达到120K步最先进模型性能的近90%。我们将在https://github.com/roysoumya/curriculum-GeneMask公开模型和代码。
英文摘要：
Gene transformer models such as Nucleotide Transformer, DNABert, and LOGO are trained to learn optimal gene sequence representations by using the Masked Language Modeling (MLM) training objective over the complete Human Reference Genome. However, the typical tokenization methods employ a basic sliding window of tokens, such as k-mers, that fail to utilize gene-centric semantics. This could result in the (trivial) masking of easily predictable sequences, leading to inefficient MLM training. Time-variant training strategies are known to improve pretraining efficiency in both language and vision tasks. In this work, we focus on using curriculum masking where we systematically increase the difficulty of masked token prediction task by using a Pointwise Mutual Information-based difficulty criterion, as gene sequences lack well-defined semantic units similar to words or sentences of NLP domain. Our proposed Curriculum Masking-based Gene Masking Strategy (CM-GEMS) demonstrates superior representation learning capabilities compared to baseline masking approaches when evaluated on downstream gene sequence classification tasks. We perform extensive evaluation in both few-shot (five datasets) and full dataset settings (Genomic Understanding Evaluation benchmark consisting of 27 tasks). Our findings reveal that CM-GEMS outperforms state-of-the-art models (DNABert-2, Nucleotide transformer, DNABert) trained at 120K steps, achieving similar results in just 10K and 1K steps. We also demonstrate that Curriculum-Learned LOGO (a 2-layer DNABert-like model) can achieve nearly 90% of the state-of-the-art model performance of 120K steps. We will make the models and codes publicly available at https://github.com/roysoumya/curriculum-GeneMask.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
