NDP：将下一分布预测作为更广泛的目标
NDP: Next Distribution Prediction as a More Broad Target
AI总结：
本文提出下一分布预测（NDP），用n-gram分布替代NTP中的独热目标以缓解目标窄化问题，在翻译、通用任务和医学领域均取得显著提升。
摘要：
基于下一词元预测（NTP）范式训练的大语言模型（LLMs）已展现出强大能力。然而，现有的NTP范式存在若干局限，尤其与推理过程中的计划性任务复杂度和错误传播有关。在我们的工作中，我们扩展了对NTP的批评，强调其局限还源于训练目标过于狭窄：即预测一个次优的独热分布。为支持这一批评，我们进行了一项预实验，将强大LLMs的输出分布视为高效的世界数据压缩。通过评估n-gram分布与独热分布同LLMs之间的相似性，我们观察到n-gram分布与LLMs的输出分布更为一致。基于这一见解，我们提出下一分布预测（NDP），使用n-gram分布替代独热目标，在不增加额外在线训练时间的情况下增强学习。我们在翻译、通用任务、语言迁移和医学领域自适应方面进行了实验。与NTP相比，NDP在翻译任务中可实现最高+2.97的COMET提升，在通用任务中平均提升+0.61，在医学领域平均提升高达+10.75。这表明解决目标窄化问题具有切实益处，为未来改进NTP的工作指明了新方向。
英文摘要：
Large language models (LLMs) trained on next-token prediction (NTP) paradigm have demonstrated powerful capabilities. However, the existing NTP paradigm contains several limitations, particularly related to planned task complications and error propagation during inference. In our work, we extend the critique of NTP, highlighting its limitation also due to training with a narrow objective: the prediction of a sub-optimal one-hot distribution. To support this critique, we conducted a pre-experiment treating the output distribution from powerful LLMs as efficient world data compression. By evaluating the similarity between the $n$-gram distribution and the one-hot distribution with LLMs, we observed that the $n$-gram distributions align more closely with the output distribution of LLMs. Based on this insight, we introduce Next Distribution Prediction (NDP), which uses $n$-gram distributions to replace the one-hot targets, enhancing learning without extra online training time. We conducted experiments across translation, general task, language transfer, and medical domain adaptation. Compared to NTP, NDP can achieve up to +2.97 COMET improvement in translation tasks, +0.61 average improvement in general tasks, and incredible +10.75 average improvement in the medical domain. This demonstrates the concrete benefits of addressing the target narrowing problem, pointing to a new direction for future work on improving NTP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
