GNNavi:利用图神经网络导航大语言模型中的信息流
GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network
AI总结:
GNNavi 提出一种基于图神经网络的参数高效微调方法,通过将标签词作为信息传播锚点并硬连接期望信息流,在仅更新 0.2%–0.5% 参数的情况下超越标准提示微调方法。
摘要:
大语言模型(LLMs)在使用带有演示的提示时展现出强大的上下文学习(ICL)能力。然而,微调对于进一步增强其适应性仍然至关重要。基于提示的微调被证明是在低数据场景下一种有效的微调方法,但对计算资源的高需求限制了其实用性。我们通过引入一种基于提示的参数高效微调(PEFT)方法来解决这一问题。GNNavi 利用了对 ICL 信息流动态的洞察,该洞察表明标签词在提示中充当信息传播的锚点。GNNavi 采用图神经网络(GNN)层,通过将期望的信息流硬连接到 GNN 中,在处理提示的过程中精确引导信息流的聚合与分布。我们在使用 GPT-2 和 Llama2 的文本分类任务上的实验表明,GNNavi 在少样本设置下仅更新 0.2% 至 0.5% 的参数,就超越了标准的基于提示的微调方法。我们将 GNNavi 与主流的 PEFT 方法(如前缀微调、LoRA 和 Adapter)在性能和效率方面进行了比较。我们的分析揭示,GNNavi 增强了信息流并确保了清晰的聚合过程。
英文摘要:
Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used. However, fine-tuning still remains crucial to further enhance their adaptability. Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality. We address this issue by introducing a prompt-based parameter-efficient fine-tuning (PEFT) approach. GNNavi leverages insights into ICL's information flow dynamics, which indicates that label words act in prompts as anchors for information propagation. GNNavi employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN. Our experiments on text classification tasks with GPT-2 and Llama2 show GNNavi surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. We compare GNNavi with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency. Our analysis reveals that GNNavi enhances information flow and ensures a clear aggregation process.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。