Duplex：一种面向具有混合专家、分组查询注意力和连续批处理的大型语言模型的设备
Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching
AI总结：
针对大型语言模型中MoE和注意力层因连续批处理导致的低算术强度和频繁DRAM访问问题，提出Duplex设备，集成高Op/B的xPU与低Op/B的Logic-PIM，根据各层Op/B动态选择处理器并提出专家与注意力协同处理以最大化利用硬件。
摘要：
大型语言模型（LLMs）因其跨多种上下文生成高质量内容的能力而兴起。为了减少其对计算资源爆炸式增长的需求，混合专家（MoE）应运而生。MoE层能够以更少的计算量利用海量参数。应用最先进的连续批处理技术提高了吞吐量；然而，这会导致MoE和注意力层中频繁的DRAM访问。我们观察到，传统计算设备在处理MoE和注意力层时存在局限性，这些层主导了总执行时间并表现出低算术强度（Op/B）。由于连续批处理导致MoE层中的Op/B波动，仅依靠针对低Op/B的设备（如存内计算（PIM）架构）来处理MoE层具有挑战性。为了应对这些挑战，我们提出了Duplex，它包含专为高Op/B定制的xPU和在单一设备内有效执行低Op/B操作的Logic-PIM。Duplex根据LLMs内每一层的Op/B选择最合适的处理器。由于MoE层的Op/B至少为1，而分组查询注意力层的Op/B值为4-8，先前的PIM架构效率不高，它们将处理单元放置在DRAM裸片内部，且仅针对极低Op/B（低于1）的操作。基于近期趋势，Logic-PIM增加了更多硅通孔（TSV）以实现DRAM裸片与逻辑裸片之间的高带宽通信，并将强大的处理单元置于逻辑裸片上，这最适合处理从几个到几十个不等的低Op/B操作。为了最大化利用xPU和Logic-PIM，我们提出了专家与注意力协同处理。
英文摘要：
Large language models (LLMs) have emerged due to their capability to generate high-quality content across diverse contexts. To reduce their explosively increasing demands for computing resources, a mixture of experts (MoE) has emerged. The MoE layer enables exploiting a huge number of parameters with less computation. Applying state-of-the-art continuous batching increases throughput; however, it leads to frequent DRAM access in the MoE and attention layers. We observe that conventional computing devices have limitations when processing the MoE and attention layers, which dominate the total execution time and exhibit low arithmetic intensity (Op/B). Processing MoE layers only with devices targeting low-Op/B such as processing-in-memory (PIM) architectures is challenging due to the fluctuating Op/B in the MoE layer caused by continuous batching. To address these challenges, we propose Duplex, which comprises xPU tailored for high-Op/B and Logic-PIM to effectively perform low-Op/B operation within a single device. Duplex selects the most suitable processor based on the Op/B of each layer within LLMs. As the Op/B of the MoE layer is at least 1 and that of the attention layer has a value of 4-8 for grouped query attention, prior PIM architectures are not efficient, which place processing units inside DRAM dies and only target extremely low-Op/B (under one) operations. Based on recent trends, Logic-PIM adds more through-silicon vias (TSVs) to enable high-bandwidth communication between the DRAM die and the logic die and place powerful processing units on the logic die, which is best suited for handling low-Op/B operations ranging from few to a few dozens. To maximally utilize the xPU and Logic-PIM, we propose expert and attention co-processing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
