arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06358cs.CLcs.AI
收藏

SHINE:一种可扩展的上下文超网络,用于在单次传递中将上下文映射到LoRA

SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass

Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Technion, NVIDIA(技术学院与NVIDIA);University of Oxford(牛津大学);School of Electronics Engineering(电子工程学院);Computer Science, Peking University(计算机科学,北京大学)

Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelberg, Haggai Maron, Muhan Zhang

AI总结:

本文提出SHINE,一种可扩展的上下文超网络,用于在单次传递中将多样且有意义的上下文映射到高质量的LoRA适配器,通过重用冻结LLM的自身参数和引入架构创新,克服了先前超网络的关键限制,以较少的参数实现了强大的表达能力。

摘要:

我们提出SHINE(可扩展的上下文超网络),一种可扩展的超网络,能够将多样且有意义的上下文映射到高质量的LoRA适配器,用于大型语言模型(LLMs)。通过在上下文超网络设计中重用冻结LLM的自身参数,并引入架构创新,SHINE克服了先前超网络的关键限制,以相对较少的参数实现了强大的表达能力。我们引入了预训练和指令微调流水线,并训练我们的超网络在单次前向传递中从多样且有意义的上下文中生成高质量的LoRA适配器。它在不进行微调的情况下更新LLM参数,并立即启用与上下文相关的复杂问答任务,而无需直接访问上下文,有效地将上下文知识转换为参数知识。我们的工作在各种任务上取得了出色的结果,相比基于SFT的LLM适应方法,大大节省了时间和计算和内存成本,并展示了良好的可扩展性潜力。我们的代码可在https://github.com/MuLabPKU/SHINE获取。

英文摘要:

We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/MuLabPKU/SHINE

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑