Hypformer：在双曲空间中全面探索高效Transformer
Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Space
AI总结：
针对双曲空间中Transformer适配研究有限的问题，提出基于Lorentz模型的Hypformer，通过定义双曲空间核心模块及线性自注意力机制，首次实现十亿规模图数据和长序列处理，验证了其有效性与可扩展性。
摘要：
双曲几何在建模复杂结构化数据（尤其是具有潜在树状和层次结构的数据）方面展现出显著潜力。尽管各类双曲神经网络在众多领域表现出色，但将Transformer适配到双曲空间的研究仍十分有限。以往尝试主要聚焦于修改Transformer中的自注意力模块，然而这些努力未能开发出完整的双曲Transformer。这主要源于：（i）双曲空间中缺乏定义明确的模块，包括线性变换层、LayerNorm层、激活函数、dropout操作等；（ii）现有双曲自注意力模块的时间复杂度与输入token数量呈二次方关系，阻碍了其可扩展性。为应对这些挑战，我们提出Hypformer——一种基于双曲几何Lorentz模型的新型双曲Transformer。在Hypformer中，我们引入两个基础块，定义了Transformer在双曲空间中的核心模块。此外，我们开发了双曲空间中的线性自注意力机制，首次使双曲Transformer能够处理十亿规模的图数据和长序列输入。我们的实验结果证实了Hypformer在多个数据集上的有效性和效率，表明其作为大规模数据表示和大模型的高效可扩展解决方案的潜力。
英文摘要：
Hyperbolic geometry have shown significant potential in modeling complex structured data, particularly those with underlying tree-like and hierarchical structures. Despite the impressive performance of various hyperbolic neural networks across numerous domains, research on adapting the Transformer to hyperbolic space remains limited. Previous attempts have mainly focused on modifying self-attention modules in the Transformer. However, these efforts have fallen short of developing a complete hyperbolic Transformer. This stems primarily from: (i) the absence of well-defined modules in hyperbolic space, including linear transformation layers, LayerNorm layers, activation functions, dropout operations, etc. (ii) the quadratic time complexity of the existing hyperbolic self-attention module w.r.t the number of input tokens, which hinders its scalability. To address these challenges, we propose, Hypformer, a novel hyperbolic Transformer based on the Lorentz model of hyperbolic geometry. In Hypformer, we introduce two foundational blocks that define the essential modules of the Transformer in hyperbolic space. Furthermore, we develop a linear self-attention mechanism in hyperbolic space, enabling hyperbolic Transformer to process billion-scale graph data and long-sequence inputs for the first time. Our experimental results confirm the effectiveness and efficiency of Hypformer across various datasets, demonstrating its potential as an effective and scalable solution for large-scale data representation and large models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
