大型语言模型中是否存在针对对话优化的分词器的案例？
Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
AI总结：
本文探讨了针对聊天对话优化分词器能否降低token数量，实验表明可节省5%-10%能源且不影响原训练语料库效率。
摘要：
大型语言模型（LLMs）的计算和能源成本呈指数级增长，这由模型规模的不断扩大以及数亿用户对LLMs的大规模采用所驱动。LLM的单位成本是计算一个token。因此，分词器在模型的效率中扮演着重要角色，并且它们被精心优化以最小化其训练语料库中文本的token数量。LLMs最流行的应用之一是聊天机器人，它们与用户进行交互。一个关键的观察是，对于这些聊天机器人而言，重要的是分词器在用户文本输入和聊天机器人响应中的性能。这些很可能不同于训练语料库中的文本。因此，一个立即出现的问题是，针对聊天对话优化分词器是否存在潜在的好处。在本文中，通过使用公开可用的聊天对话语料库来重新设计不同分词器的词汇表，并评估它们在该领域的性能，从而探索了这一想法。结果表明，对话优化的分词器一致地减少了聊天对话中的token数量，这可以带来有意义的能源节省，范围在5%到10%之间，同时对原始训练语料库的分词效率影响极小甚至略有积极影响。
英文摘要：
The computational and energy costs of Large Language Models (LLMs) have increased exponentially driven by the growing model sizes and the massive adoption of LLMs by hundreds of millions of users. The unit cost of an LLM is the computation of a token. Therefore, the tokenizer plays an important role in the efficiency of a model, and they are carefully optimized to minimize the number of tokens for the text in their training corpus. One of the most popular applications of LLMs are chatbots that interact with users. A key observation is that, for those chatbots, what is important is the performance of the tokenizer in the user text input and the chatbot responses. Those are most likely different from the text in the training corpus. So, a question that immediately arises is whether there is a potential benefit in optimizing tokenizers for chatbot conversations. In this paper, this idea is explored for different tokenizers by using a publicly available corpus of chatbot conversations to redesign their vocabularies and evaluate their performance in this domain. The results show that conversation-optimized tokenizers consistently reduce the number of tokens in chatbot dialogues, which can lead to meaningful energy savings, in the range of 5% to 10% while having minimal or even slightly positive impact on tokenization efficiency for the original training corpus.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
