arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.05448cs.LGcs.CL
收藏

任务多样性缩短了ICL平台期

Task Diversity Shortens the ICL Plateau

Harvard University(哈佛大学);Samsung Research(三星研究所);KRAFTON;UC Berkeley(加州大学伯克利分校);UCLA(加州大学洛杉矶分校)

Jaeyeon Kim, Sehyun Kwon, Joo Young Choi, Jongho Park, Jaewoong Cho, Jason D. Lee, Ernest K. Ryu

AI总结:

研究发现在多个多样的ICL任务上同时训练能缩短损失平台期,使任务更易学习,表明语言模型大规模训练的成功也得益于自然语言数据多样性带来的更易优化。

摘要:

上下文学习(ICL)描述了语言模型基于一组输入示例及随后的查询生成输出的能力。为了理解这一显著能力,研究人员研究了简化的风格化模型。这些研究一致观察到了漫长的损失平台期,在此期间模型表现出极小的改进,随后是突然的快速学习激增。在这项工作中,我们揭示了在多个多样的ICL任务上同时训练可以缩短损失平台期,使每个任务更容易学习。这一发现令人惊讶,因为它与自然直觉相矛盾,即多个ICL任务的综合复杂性会延长学习过程,而不是缩短它。我们的结果表明,近期语言模型大规模训练的成功可能不仅归因于规模数据的丰富性,还归因于自然语言训练数据的多样性所引发的更易优化(训练)。

英文摘要:

In-context learning (ICL) describes a language model's ability to generate outputs based on a set of input demonstrations and a subsequent query. To understand this remarkable capability, researchers have studied simplified, stylized models. These studies have consistently observed long loss plateaus, during which models exhibit minimal improvement, followed by a sudden, rapid surge of learning. In this work, we reveal that training on multiple diverse ICL tasks simultaneously shortens the loss plateaus, making each task easier to learn. This finding is surprising as it contradicts the natural intuition that the combined complexity of multiple ICL tasks would lengthen the learning process, not shorten it. Our result suggests that the recent success in large-scale training of language models may be attributed not only to the richness of the data at scale but also to the easier optimization (training) induced by the diversity of natural language training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑