基于模态相对预训练的文本到代码生成
Text-to-Code Generation with Modality-relative Pre-training
AI总结:
本文提出在文本到代码生成任务中,通过模态相对训练目标分离自然语言与编程语言的嵌入空间,以避免语义混淆,在多个骨干模型和测试集上取得一致性能提升。
摘要:
大型预训练语言模型近期已被扩展并成功应用于编程语言任务,通常通过对严格自然语言模型进行进一步预训练来实现——其中训练序列通常同时包含自然语言和(线性化的)编程语言。此类方法实际上将序列的两种模态映射到同一个嵌入空间中。然而,编程语言关键字(例如“while”)通常具有非常严格定义的语义。因此,从其自然语言用法中的迁移学习未必对其代码应用有益,反之亦然。假设已有一个预训练语言模型,在本研究中,我们探讨如何根据序列标记所属的模态对其进行不同的适配和表示,以最终有利于下游任务。我们实验了在进一步模型预训练期间,通过模态相对训练目标来分离模态间的嵌入空间。我们专注于文本到代码生成,并在两个骨干模型和两个测试集上观察到一致的改进,同时测量了 pass@$k$ 和一种新颖的增量变体。
英文摘要:
Large pre-trained language models have recently been expanded and applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model--where training sequences typically contain both natural and (linearised) programming language. Such approaches effectively map both modalities of the sequence into the same embedding space. However, programming language keywords (e.g. "while") often have very strictly defined semantics. As such, transfer learning from their natural language usage may not necessarily be beneficial to their code application and vise versa. Assuming an already pre-trained language model, in this work we investigate how sequence tokens can be adapted and represented differently, depending on which modality they belong to, and to the ultimate benefit of the downstream task. We experiment with separating embedding spaces between modalities during further model pre-training with modality-relative training objectives. We focus on text-to-code generation and observe consistent improvements across two backbone models and two test sets, measuring pass@$k$ and a novel incremental variation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。