arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.03295cs.CLcs.AI
收藏

用于语言建模的N-gram预测与词差分表示

N-gram Prediction and Word Difference Representations for Language Modeling

Handong Global University(韩东国际大学)

DongNyeong Heo, Daniela Noemi Rim, Heeyoul Choi

AI总结:

本文针对因果语言建模中下一词预测过度关注局部依赖的问题,提出N-gram预测框架和词差分表示,并通过集成未来N词预测结果提升下一词预测质量,在CLM和NMT任务上验证了其显著优势。

摘要:

因果语言建模(CLM)是支撑近期大语言模型(LLMs)取得显著成功的基础框架。尽管其取得了成功,但下一词预测的训练方法存在潜在风险,可能导致模型过度关注句子内部的局部依赖关系。虽然已有研究引入同时预测未来N个词的方法,但这些方法主要应用于掩码语言建模(MLM)和神经机器翻译(NMT)等任务。在本研究中,我们为CLM任务引入了一个简单的N-gram预测框架。此外,在N-gram预测框架的基础上,我们引入词差分表示(WDR)作为模型训练过程中的替代性和上下文化的目标表示。为了进一步提高下一词预测的质量,我们提出了一种集成方法,该方法融合了未来N个词的预测结果。在涵盖CLM和NMT任务的多个基准数据集上进行的实证评估表明,我们提出的方法相较于传统CLM具有显著优势。

英文摘要:

Causal language modeling (CLM) serves as the foundational framework underpinning remarkable successes of recent large language models (LLMs). Despite its success, the training approach for next word prediction poses a potential risk of causing the model to overly focus on local dependencies within a sentence. While prior studies have been introduced to predict future N words simultaneously, they were primarily applied to tasks such as masked language modeling (MLM) and neural machine translation (NMT). In this study, we introduce a simple N-gram prediction framework for the CLM task. Moreover, we introduce word difference representation (WDR) as a surrogate and contextualized target representation during model training on the basis of N-gram prediction framework. To further enhance the quality of next word prediction, we propose an ensemble method that incorporates the future N words' prediction results. Empirical evaluations across multiple benchmark datasets encompassing CLM and NMT tasks demonstrate the significant advantages of our proposed methods over the conventional CLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑