Geotokens与Geotransformers
Geotokens and Geotransformers
AI总结：
本文提出geotokens，将输入token与地理位置关联，并借鉴RoPE设计适用于球面坐标的位置编码方法，以平衡现实距离与嵌入空间距离。
摘要：
在transformer架构中，位置编码主要为输入token提供序列感。尽管原始transformer论文的方法在一般语言处理任务中显示出令人满意的结果，但已有新的提案（如Rotary Position Embedding (RoPE)）用于进一步改进。本文提出了geotokens，作为transformers的输入组件，每个都与特定的地理位置相关联。与典型的语言序列不同，对于这些token，顺序不如地理坐标本身重要。为了在此上下文中表示相对位置，并在现实世界距离与嵌入空间距离之间保持平衡，我们设计了一种借鉴RoPE结构但专为球面坐标定制的位置编码方法。
英文摘要：
In transformer architectures, position encoding primarily provides a sense of sequence for input tokens. While the original transformer paper's method has shown satisfactory results in general language processing tasks, there have been new proposals, such as Rotary Position Embedding (RoPE), for further improvement. This paper presents geotokens, input components for transformers, each linked to a specific geological location. Unlike typical language sequences, for these tokens, the order is not as vital as the geographical coordinates themselves. To represent the relative position in this context and to keep a balance between the real world distance and the distance in the embedding space, we design a position encoding approach drawing from the RoPE structure but tailored for spherical coordinates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
