Transolver真的需要Transformer吗？
Does Transolver really need a Transformer?
AI总结：
本研究通过消融实验和理论分析证明，Transolver中的注意力机制是冗余的，仅切片/反切片与逐点MLP即可实现通用逼近，并提出了高效的flashslice内核实现。
AI中文摘要：
广泛使用的Transolver系列神经算子基于物理注意力机制，该机制将非结构化网格的点软分配到少量切片中，对生成的令牌应用自注意力，并将结果广播回各点。我们进行了全面的实证和理论分析，以阐明决定模型性能的机制。为此，我们在九个具有挑战性的三维流体动力学基准上进行了仔细的消融实验，发现用常量线性映射替换令牌注意力不会影响精度。因此，Transolver根本不需要Transformer。然而，移除全局混合（切片/反切片）或仅执行一次会导致性能崩溃。我们利用平均神经算子的理论来解释并证实我们的发现，表明仅切片/反切片与逐点MLP结合就足以实现对连续算子的通用逼近，而注意力在此背景下是冗余的。最后，我们提供了一种新颖的FlashAttention风格的高效实现，用于Transolver的关键切片/反切片模块。该flashslice内核在点上流式执行切片和反切片，无需物化沉重的切片权重张量，同时以浮点误差精度复现最佳现有实现。同时，它带来了非常显著的内存和计算节省，尤其是在切片数量较大时。
英文摘要：
The widely used Transolver family of neural operators is based on physics-attention, which softly assigns the points of an unstructured mesh to a small number of slices, applies self-attention among the resulting tokens, and broadcasts the result back to the points. We provide a comprehensive empirical and theoretical analysis to elucidate the mechanisms which are responsible for model performance. To this end, we perform careful ablations on a challenging suite of nine 3D fluid dynamics benchmarks to find that replacing token attention with a constant linear map does not affect the accuracy. Thus, Transolver does not need a Transformer at all. However, removing the global mixing (slicing/deslicing) or doing it only once leads to performance collapse. We leverage the theory of averaging neural operators to explain and corroborate our findings by showing that just slicing/deslicing, in conjunction with pointwise MLPs, already suffices for universal approximation of continuous operators and attention is redundant in this context. Finally, we provide a novel FlashAttention-style efficient implementation of the key slicing/deslicing module of Transolver. This flashslice kernel streams slice and deslice over the points without materializing the heavy slice-weight tensor, while reproducing the best available implementation to floating point error. At the same time, it leads to very significant memory and compute savings, particularly at large slice counts.
