T-FREE：通过稀疏表示实现无分词器的子词生成式 LLM，以获得内存高效的嵌入
T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
AI总结：
本文提出 T-FREE，利用字符三元组稀疏激活直接嵌入单词，无需参考语料库，可将嵌入与输出头层参数削减逾 85%，并提升跨语言迁移性能。
摘要：
分词器对于在大型语言模型中编码信息至关重要，但其发展近期已停滞，且存在固有弱点。主要局限包括计算开销、词汇使用低效，以及不必要的庞大嵌入层和输出头层。此外，其性能偏向参考语料库，导致对代表性不足语言的有效性降低。为解决这些问题，我们提出 T-FREE，它通过字符三元组上的稀疏激活模式直接嵌入单词，并且不需要参考语料库。T-FREE 天然利用形态相似性，并支持对嵌入层进行强力压缩。在详尽的实验评估中，我们在这些层上实现了超过 85% 的参数削减，同时取得具有竞争力的下游性能。此外，T-FREE 在跨语言迁移学习中表现出显著提升。
英文摘要：
Tokenizers are crucial for encoding information in Large Language Models, but their development has recently stagnated, and they contain inherent weaknesses. Major limitations include computational overhead, ineffective vocabulary use, and unnecessarily large embedding and head layers. Additionally, their performance is biased towards a reference corpus, leading to reduced effectiveness for underrepresented languages. To remedy these issues, we propose T-FREE, which directly embeds words through sparse activation patterns over character triplets, and does not require a reference corpus. T-FREE inherently exploits morphological similarities and allows for strong compression of embedding layers. In our exhaustive experimental evaluation, we achieve competitive downstream performance with a parameter reduction of more than 85% on these layers. Further, T-FREE shows significant improvements in cross-lingual transfer learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
