arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.13035cs.CLcs.LG
收藏

TACO-RL:基于强化学习的任务感知提示压缩优化

TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learning

Microsoft(微软公司)

Shivam Shandilya, Menglin Xia, Supriyo Ghosh, Huiqiang Jiang, Jue Zhang, Qianhui Wu, Victor Rühle

AI总结:

针对现有提示压缩方法忽略任务信息的问题,提出基于强化学习的任务感知压缩方法,利用REINFORCE算法引导压缩,在文本摘要、问答和代码摘要任务上性能提升8%-189%。

摘要:

随着GPT-4等大型语言模型(LLM)在各类应用中的日益普及,为达到最佳性能所需的提示(prompt)规模急剧增大,进而带来了计算效率方面的挑战。提示压缩旨在通过最小化输入令牌(token)数量来降低推理成本,同时不损害任务性能。然而,现有的提示压缩技术要么依赖于信息熵等次优指标,要么将其建模为无法捕捉任务特定信息的任务无关令牌分类问题。为解决这些问题,我们提出了一种新颖且高效的基于强化学习(RL)的任务感知提示压缩方法。为满足低延迟要求,我们利用现有的基于Transformer编码器的令牌分类模型,同时使用轻量级REINFORCE算法,以任务特定的奖励信号引导学习过程。我们在三个多样且具有挑战性的任务上评估了所提方法的性能,包括文本摘要、问答和代码摘要。我们证明,与最先进的压缩技术相比,我们的RL引导压缩方法在满足相同压缩率和延迟要求的同时,在这三个场景中将任务性能提升了8%至189%。

英文摘要:

The increasing prevalence of large language models (LLMs) such as GPT-4 in various applications has led to a surge in the size of prompts required for optimal performance, leading to challenges in computational efficiency. Prompt compression aims to reduce the inference cost by minimizing input tokens without compromising on the task performance. However, existing prompt compression techniques either rely on sub-optimal metrics such as information entropy or model it as a task-agnostic token classification problem that fails to capture task-specific information. To address these issues, we propose a novel and efficient reinforcement learning (RL) based task-aware prompt compression method. To ensure low latency requirements, we leverage existing Transformer encoder-based token classification model while guiding the learning process with task-specific reward signals using lightweight REINFORCE algorithm. We evaluate the performance of our method on three diverse and challenging tasks including text summarization, question answering and code summarization. We demonstrate that our RL-guided compression method improves the task performance by 8% - 189% across these three scenarios over state-of-the-art compression techniques while satisfying the same compression rate and latency requirements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑