通过质量感知的分词解锁噪声真实世界语料用于基础模型预训练
Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization
AI总结：
本研究提出质量感知分词方法，通过双层优化、强化学习和自适应参数学习机制，提升基础模型在基因组和金融领域的预训练效果。
摘要：
当前的分词方法在处理序列数据时没有考虑信号质量，限制了其在噪声真实世界语料上的有效性。我们提出了QA-Token（质量感知分词），该方法直接将数据可靠性纳入词汇构建中。我们做出了三个关键贡献：（i）一种双层优化公式，联合优化词汇构建和下游性能；（ii）一种强化学习方法，通过质量感知奖励学习合并策略，并保证收敛性；（iii）一种通过Gumbel-Softmax松弛实现的自适应参数学习机制，用于端到端优化。我们的实验评估显示了一致的改进：基因组（在变异检测中比BPE提升6.7个百分点的F1值），金融（夏普比率提升30%）。在基础规模上，我们对包含1.7万亿碱基对的预训练语料进行分词，并在不增加推理开销的情况下实现了最先进的病原体检测（94.53 MCC），同时将token数量减少了15%。我们解锁了覆盖百亿碱基对的基因组序列和数百TB金融时间序列的真实世界语料，用于基础模型训练。
英文摘要：
Current tokenization methods process sequential data without accounting for signal quality, limiting their effectiveness on noisy real-world corpora. We present QA-Token (Quality-Aware Tokenization), which incorporates data reliability directly into vocabulary construction. We make three key contributions: (i) a bilevel optimization formulation that jointly optimizes vocabulary construction and downstream performance, (ii) a reinforcement learning approach that learns merge policies through quality-aware rewards with convergence guarantees, and (iii) an adaptive parameter learning mechanism via Gumbel-Softmax relaxation for end-to-end optimization. Our experimental evaluation demonstrates consistent improvements: genomics (6.7 percentage point F1 gain in variant calling over BPE), finance (30% Sharpe ratio improvement). At foundation scale, we tokenize a pretraining corpus comprising 1.7 trillion base-pairs and achieve state-of-the-art pathogen detection (94.53 MCC) while reducing token count by 15%. We unlock noisy real-world corpora, spanning petabases of genomic sequences and terabytes of financial time series, for foundation model training with zero inference overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
