单细胞转录组学中掩码重建Transformer的缩放定律
Scaling Laws for Masked-Reconstruction Transformers on Single-Cell Transcriptomics
AI总结:
本研究首次系统探索单细胞RNA测序数据上掩码重建Transformer的缩放行为,发现数据充足时存在幂律缩放定律,数据稀缺时缩放可忽略,并指出数据-参数比是关键决定因素。
摘要:
神经缩放定律——损失、模型大小和数据之间的幂律关系——已在语言和视觉Transformer中得到广泛记录,但它们在单细胞基因组学中的存在性仍未得到充分探索。我们首次系统研究了在单细胞RNA测序(scRNA-seq)数据上训练的掩码重建Transformer的缩放行为。使用CELLxGENE Census的表达谱,我们构建了两种实验设置:数据丰富设置(512个高度可变基因,200,000个细胞)和数据有限设置(1,024个基因,10,000个细胞)。在参数数量跨越三个数量级(533到3.4×10^8个参数)的七种模型大小上,我们将参数化缩放定律拟合到验证均方误差(MSE)。数据丰富设置表现出清晰的幂律缩放,不可约损失下限c约为1.44,而数据有限设置显示出可忽略的缩放,表明当数据稀缺时模型容量不是约束条件。这些结果确立了类似于自然语言处理中观察到的缩放定律在单细胞转录组学中确实存在(当数据充足时),并确定了数据-参数比是缩放行为的关键决定因素。将数据丰富渐近下限初步转换为信息论单位,估计每个掩码基因位置约2.30比特熵。我们讨论了对单细胞基础模型设计的启示,并概述了完善该熵估计所需的额外测量。
英文摘要:
Neural scaling laws -- power-law relationships between loss, model size, and data -- have been extensively documented for language and vision transformers, yet their existence in single-cell genomics remains largely unexplored. We present the first systematic study of scaling behaviour for masked-reconstruction transformers trained on single-cell RNA sequencing (scRNA-seq) data. Using expression profiles from the CELLxGENE Census, we construct two experimental regimes: a data-rich regime (512 highly variable genes, 200,000 cells) and a data-limited regime (1,024 genes, 10,000 cells). Across seven model sizes spanning three orders of magnitude in parameter count (533 to 3.4 x 10^8 parameters), we fit the parametric scaling law to validation mean squared error (MSE). The data-rich regime exhibits clear power-law scaling with an irreducible loss floor of c ~ 1.44, while the data-limited regime shows negligible scaling, indicating that model capacity is not the binding constraint when data are scarce. These results establish that scaling laws analogous to those observed in natural language processing do emerge in single-cell transcriptomics when sufficient data are available, and they identify the data-to-parameter ratio as a critical determinant of scaling behaviour. A preliminary conversion of the data-rich asymptotic floor to information-theoretic units yields an estimate of approximately 2.30 bits of entropy per masked gene position. We discuss implications for the design of single-cell foundation models and outline the additional measurements needed to refine this entropy estimate.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。