深度学习算子的验证提升
Verified Lifting of Deep learning Operators
AI总结:
本文提出一种深度学习算子验证提升框架,结合符号执行、语法引导合成、SMT验证和e-graph简化,从Triton底层实现自动合成并形式化验证高层数学公式,弥合底层实现与高层抽象之间的鸿沟。
摘要:
深度学习算子是现代深度学习框架的基础组成部分。随着对定制算子需求的不断增长,开发者自行创建算子已变得越来越普遍。然而,由于硬件特定的优化以及对数值稳定性的需求,设计和实现算子既复杂又容易出错。当前迫切需要能够总结现有算子及用户自定义算子功能的工具。为填补这一空白,本文提出了一个用于深度学习算子验证提升(verified lifting)的新框架,该框架从底层实现中合成高层数学公式。我们的方法结合了符号执行、语法引导合成(syntax-guided synthesis)以及基于SMT的验证,以生成可读且经过形式化验证的数学公式。在合成阶段,我们采用自顶向下与自底向上相结合的策略来高效探索庞大的搜索空间;在验证阶段,我们设计了不变式合成模式,并利用SMT求解器验证所推导摘要的正确性;在简化阶段,我们使用基于e-graph的技术和自定义规则,将复杂公式还原为其自然、直观的形式。我们在来自真实世界、用Triton实现的深度学习算子数据集上进行了评估,结果表明,与现有技术相比,我们的方法在合成和验证方面具有显著的有效性。该框架弥合了底层实现与高层抽象之间的鸿沟,提升了深度学习算子开发中的可理解性和可靠性。
英文摘要:
Deep learning operators are fundamental components of modern deep learning frameworks. With the growing demand for customized operators, it has become increasingly common for developers to create their own. However, designing and implementing operators is complex and error-prone, due to hardware-specific optimizations and the need for numerical stability. There is a pressing need for tools that can summarize the functionality of both existing and user-defined operators. To address this gap, this work introduces a novel framework for the verified lifting of deep learning operators, which synthesizes high-level mathematical formulas from low-level implementations. Our approach combines symbolic execution, syntax-guided synthesis, and SMT-based verification to produce readable and formally verified mathematical formulas. In synthesis, we employ a combination of top-down and bottom-up strategies to explore the vast search space efficiently; In verification, we design invariant synthesis patterns and leverage SMT solvers to validate the correctness of the derived summaries; In simplification, we use egraph-based techniques with custom rules to restore complex formulas to their natural, intuitive forms. Evaluated on a dataset of deep learning operators implemented in Triton from the real world, our method demonstrates the effectiveness of synthesis and verification compared to existing techniques. This framework bridges the gap between low-level implementations and high-level abstractions, improving understanding and reliability in deep learning operator development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。