基于动态块稀疏注意力的高效多样本上下文学习
Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention
AI总结:
针对多样本 ICL 推理成本高的问题,本文提出免训练的 Dynamic Block-Sparse Attention 框架,结合块稀疏注意力与缓存演示组检索,在保持最佳方法平均准确率逾95%的同时达到接近微调的延迟。
摘要:
多样本上下文学习(many-shot in-context learning)近期已展现出作为微调替代方案的潜力,其主要优势在于同一个模型可服务于多个任务。然而,这将计算负担从训练阶段转移到了推理阶段,导致多样本 ICL 在实际应用中的部署难以证明其合理性。如果为每个推理样本检索定制的演示集,这一成本还会进一步增加。我们提出了 Dynamic Block-Sparse Attention(动态块稀疏注意力),这是一个用于基于检索的多样本上下文学习的免训练框架。通过将精心设计的块稀疏注意力与缓存演示组的检索相结合,我们实现了与微调相当的单样本延迟,同时在强大的 ICL 和微调基线中,平均保持了最佳方法准确率的 95% 以上。我们希望这将进一步推动多样本 ICL 的大规模部署。
英文摘要:
Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, a training-free framework for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average >95% of the best method's accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。