arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.15518cs.CL
收藏

Squid:长上下文作为节能设备端语言模型的新模态

Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Models

Nexa AI

Wei Chen, Zhiyuan Li, Shuo Xin, Yihao Wang

AI总结:

研究针对设备端语言模型处理长上下文能耗高、延迟大的问题,提出Dolphin架构,用紧凑解码器提炼上下文信息,结合图像嵌入投影仪编码长文本,经实证评估,该方法提升了能源效率,降低了延迟,为设备端语言模型发展做贡献。

摘要:

本文提出了Dolphin,一种用于语言模型中长上下文节能处理的新型解码器-解码器架构。我们的方法解决了设备端模型中固有的显著能耗和延迟挑战。Dolphin采用一个紧凑的5亿参数解码器将大量上下文信息提炼到一个内存嵌入中,大幅减少主70亿参数解码器模型的输入长度。受视觉语言模型启发,我们重新利用图像嵌入投影仪对长文本上下文进行编码,将扩展上下文有效地视为一种独特的模态。实证评估表明,与传统的全长上下文处理方法相比,能源效率提高了10倍,延迟降低了5倍,且不损失响应质量。我们的工作有助于为设备端应用开发更可持续、可扩展的语言模型,满足资源受限环境中对节能和响应式人工智能技术的迫切需求,同时保持理解长上下文的准确性。这项研究对自然语言处理的更广泛领域有影响,特别是在资源受限环境下的高效模型设计领域。通过在边缘设备上实现更复杂的人工智能能力,Dolphin为计算资源稀缺的广泛应用中的高级语言处理铺平了道路。Dolphin模型可通过此https URL公开获取。

英文摘要:

This paper presents Dolphin, a novel decoder-decoder architecture for energy-efficient processing of long contexts in language models. Our approach addresses the significant energy consumption and latency challenges inherent in on-device models. Dolphin employs a compact 0.5B parameter decoder to distill extensive contextual information into a memory embedding, substantially reducing the input length for the primary 7B parameter decoder model. Inspired by vision-language models, we repurpose the image embedding projector to encode long textual contexts, effectively treating extended context as a distinct modality. This innovative method enables processing of substantially longer contexts without the typical computational overhead associated with extended input sequences. Empirical evaluations demonstrate a 10-fold improvement in energy efficiency and a 5-fold reduction in latency compared to conventional full-length context processing methods without losing quality of the response. Our work contributes to the development of more sustainable and scalable language models for on-device applications, addressing the critical need for energy-efficient and responsive AI technologies in resource-constrained environments while maintaining the accuracy to understand long contexts. This research has implications for the broader field of natural language processing, particularly in the domain of efficient model design for resource-limited settings. By enabling more sophisticated AI capabilities on edge devices, Dolphin paves the way for advanced language processing in a wide range of applications where computational resources are at a premium. The Dolphin model is publicly available at https://huggingface.co/NexaAIDev/Dolphin.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑