MAGNET：利用基于自适应梯度的分词技术改善语言模型的多语言公平性
MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
AI总结：
本文提出MAGNET，一种基于自适应梯度的多语言分词方法，通过语言文字特定的边界预测器模块化架构减少非拉丁文字的过度分割，在降低分割差异的同时加快语言建模并提升下游任务实用性。
摘要：
在多语言环境中，非拉丁文字和低资源语言在语言模型的实用性、效率和成本方面通常处于劣势。具体而言，先前的研究报告了当前分词算法引入非拉丁文字语言的多种建模偏差，主要表现之一是过度分割。在这项工作中，我们提出了MAGNET；多语言自适应基于梯度的分词方法，通过基于自适应梯度的子词分词来减少过度分割。MAGNET通过模型内部的子模块学习预测序列中字节token之间的分割边界，这些子模块充当内部边界预测器（分词器）。先前的基于梯度的分词方法旨在通过在训练期间集成单个边界预测器，并通过随机重参数化与下一token预测目标一起进行端到端优化，从而实现序列间的均匀压缩。然而，这种方法在多语言环境中仍会导致非拉丁文字语言过度分割。相比之下，MAGNET提供了一种可定制的架构，其中字节级序列通过特定语言文字的预测器进行路由，每个预测器针对其各自的语言文字进行优化。与先前的方法相比，这种模块化设计在不同语言文字之间强制实现了公平的分割粒度。通过大量实验，我们证明MAGNET除了能减少分割差异外，还能实现更快的语言建模并提升下游实用性。
英文摘要：
In multilingual settings, non-Latin scripts and low-resource languages are usually disadvantaged in terms of language models' utility, efficiency, and cost. Specifically, previous studies have reported multiple modeling biases that the current tokenization algorithms introduce to non-Latin script languages, the main one being over-segmentation. In this work, we propose MAGNET; multilingual adaptive gradient-based tokenization to reduce over-segmentation via adaptive gradient-based subword tokenization. MAGNET learns to predict segment boundaries between byte tokens in a sequence via sub-modules within the model, which act as internal boundary predictors (tokenizers). Previous gradient-based tokenization methods aimed for uniform compression across sequences by integrating a single boundary predictor during training and optimizing it end-to-end through stochastic reparameterization alongside the next token prediction objective. However, this approach still results in over-segmentation for non-Latin script languages in multilingual settings. In contrast, MAGNET offers a customizable architecture where byte-level sequences are routed through language-script-specific predictors, each optimized for its respective language script. This modularity enforces equitable segmentation granularity across different language scripts compared to previous methods. Through extensive experiments, we demonstrate that in addition to reducing segmentation disparities, MAGNET also enables faster language modelling and improves downstream utility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
