PaPaformer:来自预训练并行路径的语言模型
PaPaformer: Language Model from Pre-trained Parallel Paths
AI总结:
PaPaformer通过并行低维路径预训练后组合成更大模型,实现数小时内训练解码器Transformer语言模型,减少参数与训练时间并提升性能,支持任务定制。
摘要:
现代大型语言模型的训练需要越来越多的计算能力和时间。即使是较小的变体,如小型语言模型(SLMs),在最佳情况下也需要数天时间进行训练,且通常需要多个GPU。本文探索了在数小时而非数天/数周内训练和评估仅解码器Transformer语言模型的方法。我们引入了PaPaformer,一种仅解码器Transformer架构变体,其较低维度的并行路径被组合成更大的模型。论文表明,这些较低维度的路径可以使用不同类型的训练数据单独训练,然后组合成一个更大的模型。该方法提供了减少模型总参数数量和训练时间同时提升性能的选择。此外,并行路径结构的使用为定制路径以适应特定任务需求开辟了有趣的可能性。
英文摘要:
The training of modern large-language models requires an increasingly amount of computation power and time. Even smaller variants, such as small-language models (SLMs), take several days to train in the best-case scenarios, often requiring multiple GPUs. This paper explores methods to train and evaluate decoder-only transformer-based language models in hours instead of days/weeks. We introduces \textit{PaPaformer}, a decoder-only transformer architecture variant, whose lower-dimensional parallel paths are combined into larger model. The paper shows that these lower-dimensional paths can be trained individually with different types of training data and then combined into one larger model. This method gives the option to reduce the total number of model parameters and the training time with increasing performance. Moreover, the use of parallel path structure opens interesting possibilities to customize paths to accommodate specific task requirements.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。