神经机器翻译的局部字节融合
Local Byte Fusion for Neural Machine Translation
AI总结：
针对子词分词在跨语料库及多语言低资源场景下过度分割导致性能下降的问题，提出局部字节融合（LOBEF）方法，利用字节 $n$-gram 和词边界聚合局部语义信息，在多语言翻译等任务上超越传统字节模型甚至子词技术，且参数高效、训练更快。
摘要：
子词分词方案是当前NLP模型中使用的主导技术。然而，此类方案可能较为僵化，且基于一个语料库构建的分词器不能很好地适应其他平行语料库。据观察，在多语言语料库中，子词分词方案会对低资源语言进行过度分割，导致翻译性能下降。子词分词器的一个简单替代方案是基于字节的方法，即使用 UTF-8 等编码方案将输入分词为字节序列。字节词元通常以子字符粒度表示输入，即一个字符可以由多个字节词元组成的序列表示。这导致字节序列明显长于字符序列。在较低层强制聚合局部信息可以引导模型构建更高层次的语义信息。我们提出了一种用于基于字节的机器翻译的局部字节融合（LOBEF）方法——利用字节 $n$-gram 和词边界——来聚合局部语义信息。在多语言翻译、零样本跨语言迁移和领域适应上的大量实验表明，该方法相比传统的基于字节的模型有一致的提升，甚至超越了子词技术。进一步分析还表明，我们基于字节的模型参数高效，且训练速度比子词模型更快。
英文摘要：
Subword tokenization schemes are the dominant technique used in current NLP models. However, such schemes can be rigid and tokenizers built on one corpus do not adapt well to other parallel corpora. It has also been observed that in multilingual corpora, subword tokenization schemes over-segment low-resource languages leading to a drop in translation performance. A simple alternative to subword tokenizers is byte-based methods i.e. tokenization into byte sequences using encoding schemes such as UTF-8. Byte tokens often represent inputs at a sub-character granularity i.e. one character can be represented by a sequence of multiple byte tokens. This results in byte sequences that are significantly longer than character sequences. Enforcing aggregation of local information in the lower layers can guide the model to build higher-level semantic information. We propose a Local Byte Fusion (LOBEF) method for byte-based machine translation -- utilizing byte $n$-gram and word boundaries -- to aggregate local semantic information. Extensive experiments on multilingual translation, zero-shot cross-lingual transfer, and domain adaptation reveal a consistent improvement over traditional byte-based models and even over subword techniques. Further analysis also indicates that our byte-based models are parameter-efficient and can be trained faster than subword models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
