arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.04183cs.CLcs.AI
收藏

GALLa:面向改进源代码理解的图对齐大语言模型

GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding

Ant Group(蚂蚁集团);Shanghai Jiao Tong University(上海交通大学)

Ziyin Zhang, Hang Yu, Shijie Li, Peng Di, Jianguo Li, Rui Wang

AI总结:

GALLa通过图神经网络与跨模态对齐将代码结构信息作为辅助任务注入LLM微调,在保持模型无关和推理零开销的同时,于多个代码任务上一致提升基线模型性能。

摘要:

编程语言具有丰富的语义信息——例如数据流——这些信息由图表示,无法从源代码的表面形式中获得。最近的代码语言模型已扩展到数十亿参数,但仅将源代码建模为文本token,而忽略了任何其他结构信息。相反,确实编码代码结构信息的模型会对Transformer架构进行修改,从而限制了其规模以及与预训练LLM的兼容性。在这项工作中,我们通过GALLa——图对齐大语言模型——兼取两者之长。GALLa利用图神经网络和跨模态对齐技术,在微调期间将代码的结构信息作为辅助任务注入LLM中。该框架既与模型无关,也与任务无关,因为它可以应用于任何代码LLM以执行任何代码下游任务,并且仅在训练时需要来自与微调数据无关的语料库的结构图数据,而在推理时相对于基线LLM不产生任何额外成本。在五个代码任务上使用七种不同的基线LLM(规模从350M到14B不等)进行的实验验证了GALLa的有效性,证明了相对于基线的持续改进,即使对于LLaMA3和Qwen2.5-Coder等强大模型也是如此。

英文摘要:

Programming languages possess rich semantic information - such as data flow - that is represented by graphs and not available from the surface form of source code. Recent code language models have scaled to billions of parameters, but model source code solely as text tokens while ignoring any other structural information. Conversely, models that do encode structural information of code make modifications to the Transformer architecture, limiting their scale and compatibility with pretrained LLMs. In this work, we take the best of both worlds with GALLa - Graph Aligned Large Language Models. GALLa utilizes graph neural networks and cross-modal alignment technologies to inject the structural information of code into LLMs as an auxiliary task during finetuning. This framework is both model-agnostic and task-agnostic, as it can be applied to any code LLM for any code downstream task, and requires the structural graph data only at training time from a corpus unrelated to the finetuning data, while incurring no cost at inference time over the baseline LLM. Experiments on five code tasks with seven different baseline LLMs ranging in size from 350M to 14B validate the effectiveness of GALLa, demonstrating consistent improvement over the baseline, even for powerful models such as LLaMA3 and Qwen2.5-Coder.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑