arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.08698cs.LGq-bio.GN
收藏

EvoLen:基于进化的标记化方法用于DNA语言模型

EvoLen: Evolution-Guided Tokenization for DNA Language Model

University of California, San Diego(加州大学圣地亚哥分校);Washington University in St. Louis(圣路易斯华盛顿大学)

Nan Huang, Xiaoxiao Zhou, Junxia Cui, Mario Tapia-Pacheco, Tiffany Amariuta, Yang Li, Jingbo Shang

AI总结:

EvoLen通过整合进化信息优化DNA标记化,优先保留功能序列模式,提升基因组上下文区分和进化约束对齐,优于标准BPE。

摘要:

在DNA语言模型中,标记作为基本表示单元,其设计仍待深入探索。不同于自然语言,DNA缺乏固有标记边界或预定义组合规则,使其标记化成为基础建模决策而非自然指定。现有方法如字节对编码(BPE)擅长捕捉反映人类生成语言规律的标记结构,但DNA由生物功能和进化约束组织而非语言惯例。本文主张DNA标记化应优先考虑功能序列模式,如调控位点——短、重复的进化约束下通常跨物种保存。EvoLen通过进化分层与长度感知解码结合,直接整合进化信息以更好地保留位点尺度的功能序列单元。EvoLen利用跨物种进化信号分组DNA序列,分别在每组上训练BPE标记器,通过规则合并词汇表优先保留保存的模式,并应用长度感知解码与动态规划。通过受控实验,EvoLen提升了功能序列模式的保留、基因组上下文区分和与进化约束的对齐,同时在多种DNALM基准上匹配或超越标准BPE。这些结果表明标记化引入了关键归纳偏置,整合进化信息可获得更具生物意义和可解释的序列表示。

英文摘要:

Tokens serve as the basic units of representation in DNA language models (DNALMs), yet their design remains underexplored. Unlike natural language, DNA lacks inherent token boundaries or predefined compositional rules, making tokenization a fundamental modeling decision rather than a naturally specified one. While existing approaches like byte-pair encoding (BPE) excel at capturing token structures that reflect human-generated linguistic regularities, DNA is organized by biological function and evolutionary constraint rather than linguistic convention. We argue that DNA tokenization should prioritize functional sequence patterns like regulatory motifs-short, recurring segments under evolutionary constraint and typically preserved across species. We incorporate evolutionary information directly into the tokenization process through EvoLen, a tokenizer that combines evolutionary stratification with length-aware decoding to better preserve motif-scale functional sequence units. EvoLen uses cross-species evolutionary signals to group DNA sequences, trains separate BPE tokenizers on each group, merges the resulting vocabularies via a rule prioritizing preserved patterns, and applies length-aware decoding with dynamic programming. Through controlled experiments, EvoLen improves the preservation of functional sequence patterns, differentiation across genomic contexts, and alignment with evolutionary constraint, while matching or outperforming standard BPE across diverse DNALM benchmarks. These results demonstrate that tokenization introduces a critical inductive bias and that incorporating evolutionary information yields more biologically meaningful and interpretable sequence representations. Code, pretrained and fine-tuned checkpoints, and tokenizer files are available at https://github.com/HN020719/EvoLen and https://huggingface.co/EvoLenTokenizer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑