分段超越默认设置：用于最优机器翻译性能的不对称字对编码
Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance
AI总结：
本研究提出不对称字对编码以提升低资源条件下机器翻译性能，通过不同NMO设置显著提高翻译质量。
摘要：
现有的机器翻译（MT）研究通常建议为词分段模型使用单一、固定的超参数集，对称字对编码（BPE），该方法对源语言和目标语言应用相同的合并操作次数（NMO）来训练分词器。然而，我们证明这种统一方法并不能保证在不同的语言对和数据量下都能获得最优的MT性能。本研究在各种数据量和语言对上调查BPE分段配方，以评估MT系统的性能。我们发现使用不对称BPE，即源语言和目标语言具有不同的NMO，比对称方法显著提高了结果，尤其是在低资源设置（50K、100K和500K句子对）中。具体来说，不对称BPE在英语-印地语低资源设置（50K、100K和500K句子对）中分别产生了统计上显著（p<0.05）的平均增益5.32、4.46和0.7 CHRF++。我们验证了这一趋势在六个额外的语言对（英语和泰卢格语、绍纳语、挪威语、基尔克语、豪萨语和因纽特语）上，观察到在12个系统中有10个比对称BPE有统计上显著的改进。我们的发现表明，较高的NMO用于源语言（4K到32K）和较低的NMO用于目标语言（0.5K到2K）提供了最佳结果，特别是对低资源MT有益。
英文摘要：
Existing Machine Translation (MT) research often suggests a single, fixed set of hyperparameters for word segmentation models, symmetric Byte Pair Encoding (BPE), which applies the same number of merge operations (NMO) to train tokenizers for both source and target languages. However, we demonstrate that this uniform approach doesn't guarantee optimal MT performance across different language pairs and data sizes. This work investigates BPE segmentation recipes across various data volumes and language pairs to evaluate MT system performance. We find that utilizing asymmetric BPE, where the source and target languages have different NMOs, significantly improves results over the symmetric approach, especially in low-resource settings (50K, 100K, and 500K sentence pairs). Specifically, asymmetric BPE yield statistically significant ($p<0.05$) average gains of 5.32, 4.46, and 0.7 CHRF++ on English-Hindi in low-resource setups (50K, 100K, and 500K sentence pairs, respectively). We validated this trend across six additional language pairs (English and Telugu, Shona, Norwegian, Kyrgyz, Hausa, and Inuktitut), observing statistically significant improvement in 10 out of 12 systems compared to symmetric BPE. Our findings indicate a high NMO for the source (4K to 32K) and a low NMO for the target (0.5K to 2K) provides optimal results, particularly benefiting low-resource MT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
