无需对话:语言模型的异步混合方法
No Need to Talk: Asynchronous Mixture of Language Models
AI总结:
提出SMALLTALK LM异步混合语言模型训练方法,无需高带宽通信,轻量路由器按短前缀导向专家,不依赖语料聚类或元数据,同FLOPs下困惑度更低且下游任务表现更优。
摘要:
我们提出SMALLTALK LM,这是一种以近乎异步的方式训练语言模型混合体的创新方法。混合体中的每个模型专注于数据分布的不同部分,且训练各模型的节点之间无需高带宽通信。推理时,一个轻量级路由器根据短前缀将给定序列导向单个专家模型。该推理方案自然仅使用整体混合模型参数的一部分。与此前的异步大语言模型(LLM)训练工作不同,我们的路由方法不依赖完整语料库聚类或元数据访问,使其更适合实际应用。语言建模实验表明,在总训练浮点运算次数(FLOPs)相同且推理成本几乎一致的情况下,SMALLTALK LM的困惑度显著低于密集模型基线。最后,在下游评估中,我们在75%的任务上优于密集基线模型。
英文摘要:
We introduce SMALLTALK LM, an innovative method for training a mixture of language models in an almost asynchronous manner. Each model of the mixture specializes in distinct parts of the data distribution, without the need for high-bandwidth communication between the nodes training each model. At inference, a lightweight router directs a given sequence to a single expert, according to a short prefix. This inference scheme naturally uses a fraction of the parameters from the overall mixture model. Unlike prior works on asynchronous LLM training, our routing method does not rely on full corpus clustering or access to metadata, making it more suitable for real-world applications. Our experiments on language modeling demonstrate that SMALLTALK LM achieves significantly lower perplexity than dense model baselines for the same total training FLOPs and an almost identical inference cost. Finally, in our downstream evaluations we outperform the dense baseline on 75% of the tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。