arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.01332cs.AIcs.DC
收藏

张量图超优化的等式饱和

Equality Saturation for Tensor Graph Superoptimization

MIT(麻省理工学院);CSAIL(计算机科学与人工智能实验室);Google(谷歌公司);University of Washington(华盛顿大学);OctoML(OctoML公司)

Yichen Yang, Phitchaya Mangpo Phothilimtha, Yisu Remy Wang, Max Willsey, Sudip Roy, Jacques Pienaar

摘要:

深度学习框架中采用的主要优化之一是图重写。生产框架依赖于语法来决定是否应该应用重写规则以及以何种顺序应用。先前的研究表明,如果我们搜索更好的替换序列而不是依赖于算法,则可以发现更优的张量计算图。然而,我们观察到,在生产和研究框架中,现有的张量图超优化方法以顺序的方式应用替换。这种顺序搜索方法对应用替换的顺序很敏感,并且通常只探索等价图的指数空间的一小部分。本文提出了一种新的张量图超优化技术,利用等式饱和度一次应用所有可能的替换。我们表明,我们的方法可以找到优化的图形,比最先进的图形加速高达16%,同时平均花费48倍的优化时间。

英文摘要:

One of the major optimizations employed in deep learning frameworks is graph rewriting. Production frameworks rely on heuristics to decide if rewrite rules should be applied and in which order. Prior research has shown that one can discover more optimal tensor computation graphs if we search for a better sequence of substitutions instead of relying on heuristics. However, we observe that existing approaches for tensor graph superoptimization both in production and research frameworks apply substitutions in a sequential manner. Such sequential search methods are sensitive to the order in which the substitutions are applied and often only explore a small fragment of the exponential space of equivalent graphs. This paper presents a novel technique for tensor graph superoptimization that employs equality saturation to apply all possible substitutions at once. We show that our approach can find optimized graphs with up to 16% speedup over state-of-the-art, while spending on average 48x less time optimizing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑