arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.22100cs.CLcs.LG
收藏

文本新内容所在的补丁:字节级模型应将计算资源分配到何处的压缩视角

When the next byte's type is fixed: why entropy patching skips computed values under tight budgets

Nicolás Vera Zúñiga

AI总结:

该研究针对字节级模型在严格补丁预算下对数学计算结果准确率下降的问题,提出新颖性补丁策略,在不同参数规模下显著提升数学任务的计算结果与最终答案准确率,且在数学外场景有适配性。

摘要:

字节级语言模型(如字节隐式Transformer(Byte Latent Transformer, BLT))会针对每个补丁运行一次大型全局模型,并在小型模型的下一字节熵较高的位置启动补丁。该规则很少在嵌入式计算结果(如解题过程中“=”后的数字)处启动补丁,且在严格的补丁预算下,模型对这类结果的准确率会大幅下降。受LZ77风格压缩的启发,我们提出了新颖性补丁策略:在后续字节不太可能重复近期上下文的位置启动补丁。该规则是一个基于前两个字节的查找表,可在原始文本上几秒内完成拟合,无需标签、参考模型或损失函数。在1.9 GB数学语料库上从头开始训练的BLT风格模型中,当补丁率匹配为~11%(每组3个随机种子)时,该策略在100万参数规模下无明显效果;在1200万参数规模下,最终答案准确率从47%提升至72%;在5000万参数规模下,计算结果准确率从21%提升至57%,最终答案准确率从60%提升至90%;在原始预训练语料库上拟合的查找表仍能保持该增益。在数学领域之外,该优势会缩小:新颖性补丁会降低散文的每字节比特数,且在源代码和合成执行轨迹上与熵表现相当。对边界位置的分析解释了这一模式:计算结果对于补丁模型具有普通意外值,因此寻求意外值的规则会忽略它们;熵规则在高意外值字节上集中补丁启动的程度最强(边界富集度为1.61),且在计算结果上表现最差,而最优规则的富集度接近随机水平。新颖性策略在新内容始于计算附近的片段起始处时有效,如书面数学内容;学习型分块器(H-Net)可在数学和代码中自主发现此类结构。我们还报告了未奏效的尝试:将类型化不确定性作为训练规则、在训练后切换规则以及上下文感知新颖性预测器。

英文摘要:

The Byte Latent Transformer (BLT) runs a large model once per patch and starts patches where a small model's next-byte entropy is high. Under tight budgets this rule skips computed values, such as the number after "= " in math. On BLT-1B at inference we trace the miss to the cue before a value: when the preceding bytes fix the next byte's type, the patcher's entropy is capped near log2 10 bits, close to a 10% budget's threshold. Entropy then starts a patch at 0% of values after a dollar sign and 19% after "= "; writing "is " instead of "= " raises the 19% to 66%. Version 1's two-byte lookup table, motivated by LZ novelty, acts as a learned detector of cues that tend to precede new values in its fitting text: on the solutions rewritten as prose it covers every value after a dollar sign and none after a plain space, and its margin over entropy falls from +42.1 to -8.9 points, partly because prose gives entropy more starts per value. In BLT-style models trained from scratch (1M-50M parameters, 128-byte context, 3 seeds per rule) and scored by teacher-forced exact match on answer spans, the table beats entropy at 50M by 36.8 points on computed results and 29.6 on final answers, which mostly copy the last computed result. A hand-written marker rule is not distinguishable from the table at 1M and 12M; its 50M control was not run. At 1M and 12M the table ran on a T4 GPU in fp16 and the baselines on an A40 in bf16; at 50M all ran on the A40. End to end, no 1M-50M model solves more than 2 of 100 GSM8K problems. A label-free probe of BLT-1B's patcher finds prose values, mostly after a dollar sign, but costs final answers. In six domains, no rule we tested beat entropy on exact match of whole words and numbers in more than one domain. The positive findings concern tight budgets on cue-formatted text. This version corrects version 1; the work is closed, with no venue intended.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑