arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.15569cs.CL
收藏

文本图上大型语言模型的高效微调与推理

Efficient Tuning and Inference for Large Language Models on Textual Graphs

Zhejiang University(浙江大学);Rutgers University(罗格斯大学)

Yun Zhu, Yaoke Wang, Haizhou Shi, Siliang Tang

AI总结:

本文提出ENGINE,一种针对文本图的参数和内存高效微调方法,通过可调侧结构结合LLMs和GNNs,在保持最佳性能的同时显著降低训练成本,并引入缓存和动态提前退出进一步提升训练和推理速度。

摘要:

网页、电子商务和学术论文等实际应用需要建模文本图丰富的文本和拓扑信息。从业者长期以来一直采用浅层文本编码器和随后的图神经网络(GNN)来解决此问题。鉴于大型语言模型(LLMs)的最新进展,显然集成LLMs以增强文本编码可以大幅提升文本图的性能。然而,这些方法的效率构成了重大挑战。在本文中,我们提出了ENGINE,一种针对带有LLM编码器的文本图的参数和内存高效的微调方法。关键见解是通过可调的侧结构结合LLMs和GNNs,这在不损害联合模型能力的情况下显著降低了训练复杂度。在文本图上的大量实验通过实现最佳模型性能,同时与以前方法相比具有最低训练成本,证明了我们方法的有效性。此外,我们引入了具有缓存和动态提前退出功能的两个变体,以进一步提升训练和推理速度。具体而言,缓存使ENGINE的训练加速12倍,动态提前退出实现了高达5倍的更快推理,且性能下降可忽略不计(在7个数据集上最大相关下降1.17%)。我们的代码可在以下网址获取:https://github.com/ZhuYun97/ENGINE

英文摘要:

Rich textual and topological information of textual graphs need to be modeled in real-world applications such as webpages, e-commerce, and academic articles. Practitioners have been long following the path of adopting a shallow text encoder and a subsequent graph neural network (GNN) to solve this problem. In light of recent advancements in large language models (LLMs), it is apparent that integrating LLMs for enhanced textual encoding can substantially improve the performance of textual graphs. Nevertheless, the efficiency of these methods poses a significant challenge. In this paper, we propose ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs with an LLM encoder. The key insight is to combine the LLMs and GNNs through a tunable side structure, which significantly reduces the training complexity without impairing the joint model's capacity. Extensive experiments on textual graphs demonstrate our method's effectiveness by achieving the best model performance, meanwhile having the lowest training cost compared to previous methods. Moreover, we introduce two variants with caching and dynamic early exit to further enhance training and inference speed. Specifically, caching accelerates ENGINE's training by 12x, and dynamic early exit achieves up to 5x faster inference with a negligible performance drop (at maximum 1.17% relevant drop across 7 datasets). Our codes are available at: https://github.com/ZhuYun97/ENGINE

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑