奇偶感知字节对编码：提高词元化中的跨语言公平性
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
AI总结：
研究针对词元化中低资源语言词元化不合理问题，引入奇偶感知字节对编码，通过公平最大化规则平衡跨语言公平性与全局压缩，减少词元化不平等，对全局压缩和下游语言模型性能影响小。
摘要：
词元化是大多数自然语言处理管道的第一步且常被忽视。标准词元化算法依赖基于频率的目标，这有利于训练数据中占主导的语言，使低资源语言词元化不合理。为此引入奇偶感知字节对编码，能减少词元化不平等，对全局压缩率影响小且不影响下游语言模型性能。
英文摘要：
Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disproportionately longer, morphologically implausible, or even riddled with $<UNK>$ placeholders. This phenomenon ultimately amplifies computational and financial inequalities between users from different language backgrounds. To remedy this, we introduce Parity-aware Byte Pair Encoding (BPE), a variant of the widely-used BPE algorithm. At every merge step, Parity-aware BPE applies a fair-max rule that maximizes the compression gain of the currently worst-compressed language, trading a small amount of global compression for cross-lingual parity. We find empirically that Parity-aware BPE reduces tokenization inequality -- operationalized by the Gini coefficient of per-language token costs -- by up to 89% relative to Classical BPE. This comes with negligible impact on global compression rate and no evidence of systematic degradation in downstream LM performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
