arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.01252cs.AIcs.LG
收藏

通过高效分层位置表示在个人轨迹中预训练上下文位置嵌入

Pre-training Contextual Location Embeddings in Personal Trajectories via Efficient Hierarchical Location Representations

SK Telecom(SK电讯);KAIST(韩国科学技术院)

Chung Park, Taesan Kim, Junui Hong, Minsung Choi, Jaegul Choo

AI总结:

针对现有位置嵌入预训练方法处理位置数量有限、成本高的问题,提出Geo-Tokenizer与分层自回归位置模型目标,以更少参数提升下游任务性能。

摘要:

对从人类移动数据生成的位置嵌入进行预训练,已成为基于位置的服务中的流行方法。在实践中,由于在细粒度分辨率或目标区域广阔的场景下需要训练的位置数量庞大,构建位置嵌入模型的成本极高。以往研究处理的不同位置数量不足一万,无法满足现实应用的需求。为解决这一问题,我们提出了Geo-Tokenizer(地理分词器),其设计思路是将单个位置表示为不同尺度下多个网格的组合,从而高效减少待训练的位置数量。在Geo-Tokenizer中,更大尺度的网格共享更小尺度的公共网格集合,这是缩减位置词表规模的关键因素。经Geo-Tokenizer预处理的位置序列会被输入因果位置嵌入模型,以捕捉位置的时间依赖关系。该模型可动态计算目标位置的嵌入向量,其结果会随对应轨迹的不同而变化。此外,为高效预训练位置嵌入模型,我们提出了Hierarchical Auto-regressive Location Model(分层自回归位置模型)目标函数,用于有效训练Geo-Tokenizer中被分解的位置。我们在两个真实世界用户轨迹数据集上对所提出的预训练位置模型开展了实验。实验结果表明,与现有位置嵌入方法相比,我们的模型以更少的模型参数显著提升了下游任务的性能。

英文摘要:

Pre-training the embedding of a location generated from human mobility data has become a popular method for location based services. In practice, modeling the location embedding is too expensive, due to the large number of locations to be trained in situations with fine-grained resolution or extensive target regions. Previous studies have handled less than ten thousand distinct locations, which is insufficient in the real-world applications. To tackle this problem, we propose a Geo-Tokenizer, designed to efficiently reduce the number of locations to be trained by representing a location as a combination of several grids at different scales. In the Geo-Tokenizer, a grid at a larger scale shares the common set of grids at smaller scales, which is a key factor in reducing the size of the location vocabulary. The sequences of locations preprocessed with the Geo-Tokenizer are utilized by a causal location embedding model to capture the temporal dependencies of locations. This model dynamically calculates the embedding vector of a target location, which varies depending on its trajectory. In addition, to efficiently pre-train the location embedding model, we propose the Hierarchical Auto-regressive Location Model objective to effectively train decomposed locations in the Geo-Tokenizer. We conducted experiments on two real-world user trajectory datasets using our pre-trained location model. The experimental results show that our model significantly improves the performance of downstream tasks with fewer model parameters compared to existing location embedding methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑