arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.04784cs.CLcs.AI
收藏

KunlunBaize:TransformerX框架下具备多尺度卷积与多Token预测的大语言模型

KunlunBaize: LLM with Multi-Scale Convolution and Multi-Token Prediction Under TransformerX Framework

Cheng Li, Jiexiong Liu, Yixuan Chen, Yanqin Jia, Zhepeng Li

AI总结:

针对大语言模型计算效率低、梯度消失等问题,提出融合可学习密集残差连接、含多尺度卷积的TransformerX模块及多Token预测的新框架,显著提升模型性能与效率。

摘要:

大语言模型已在各类任务中展现出卓越性能,但仍面临计算效率低、梯度消失以及难以捕捉复杂特征交互等挑战。为解决这些局限,本文提出了一种新型框架。该框架整合了可学习密集残差跳跃连接机制、TransformerX模块(一种融合多尺度卷积与自适应激活函数的基于Transformer的组件)以及多Token预测交互模块。可学习密集残差连接增强了跨层的信息流动与特征捕捉能力。在TransformerX模块中,大卷积核聚合长文本片段的语义信息,小卷积则聚焦于局部词序与句法结构。自适应激活函数根据输入文本的语义特征动态调整参数,提升了模型处理多样语义表达与复杂关系的能力。多Token预测模块通过预测多个未来Token,提高了数据利用率并加快了推理速度。这些组件显著提升了大语言模型的性能与效率。

英文摘要:

Large language models have demonstrated remarkable performance across various tasks, yet they face challenges such as low computational efficiency, gradient vanishing, and difficulties in capturing complex feature interactions. To address these limitations, a novel framework has been proposed. This framework incorporates a learnable dense residual skip connection mechanism, a TransformerX module a transformer based component integrating multiscale convolution and adaptive activation functions and a multitoken prediction interaction module. The learnable dense residual connections enhance information flow and feature capture across layers. Within the TransformerX module, large convolutional kernels aggregate semantic information from extensive text segments, while smaller convolutions focus on local word order and syntactic structures. The adaptive activation function dynamically adjusts its parameters based on the semantic features of the input text, improving the model's ability to handle diverse semantic expressions and complex relationships. The multitoken prediction module boosts data utilization and accelerates inference by predicting multiple future tokens. These components significantly enhance the performance and efficiency of large language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑