并行调度自注意力机制的推广与优化
Parallel Scheduling Self-attention Mechanism: Generalization and Optimization
摘要：
在过去的几年里，自注意力力在深度学习领域，特别是在自然语言处理（NLP）领域大放异彩。其令人印象深刻的有效性，以及无处不在的实现，引起了我们的兴趣，有效地调度相应的计算的数据流到具有许多计算单元的架构，以实现并行计算。本文基于自注意力机制理论和语言模型中自注意力的最新实现，提出了一种通用的调度算法，该算法由可满足性检验（SAT）求解器求解小实例的最优调度问题，从而实现了自注意力典型计算的并行化。提出了跳过冗余计算的进一步优化策略，对于两种广泛采用的自注意力应用方案，分别实现了约25%和50%的原始计算量减少。在此基础上，我们提出了另外两种调度算法。所提出的算法是适用的，无论问题的大小，只要输入向量的数量是整除的架构中可用的计算单元的数量。由于在数学上证明算法在一般情况下的正确性的复杂性，我们通过解决特定情况下的SAT问题进行了实验来揭示它们的有效性，以及所提供的解决方案的卓越质量。
英文摘要：
Over the past few years, self-attention is shining in the field of deep learning, especially in the domain of natural language processing(NLP). Its impressive effectiveness, along with ubiquitous implementations, have aroused our interest in efficiently scheduling the data-flow of corresponding computations onto architectures with many computing units to realize parallel computing. In this paper, based on the theory of self-attention mechanism and state-of-the-art realization of self-attention in language models, we propose a general scheduling algorithm, which is derived from the optimum scheduling for small instances solved by a satisfiability checking(SAT) solver, to parallelize typical computations of self-attention. Strategies for further optimization on skipping redundant computations are put forward as well, with which reductions of almost 25% and 50% of the original computations are respectively achieved for two widely-adopted application schemes of self-attention. With the proposed optimization adopted, we have correspondingly come up with another two scheduling algorithms. The proposed algorithms are applicable regardless of problem sizes, as long as the number of input vectors is divisible to the number of computing units available in the architecture. Due to the complexity of proving the correctness of the algorithms mathematically for general cases, we have conducted experiments to reveal their validity, together with the superior quality of the solutions provided by which, by solving SAT problems for particular instances.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
