GAEA: 一种具备地理位置感知能力的对话式助手
GAEA: A Geolocation Aware Conversational Assistant
AI总结：
针对现有图像地理定位模型缺乏位置理解与对话能力、大型多模态模型在该专业任务表现不佳的问题，研究提出对话模型GAEA，构建了配套的大规模数据集GAEA-1.4M与评估基准GAEA-Bench，性能显著优于现有主流多模态模型。
摘要：
图像地理定位是指AI模型传统上预测图像精确GPS坐标的任务，这是一项具有挑战性的任务，拥有众多下游应用。然而用户无法利用该模型拓展GPS坐标之外的知识，该模型既缺乏对位置的理解，也不具备与用户交流的对话能力。近年来，随着专有及开源的大型多模态模型（LMMs）取得巨大进展，研究人员尝试通过LMMs进行图像地理定位，但相关问题仍未得到解决；除通用任务外，在地理定位这类更专业的下游任务中，LMMs的表现仍很吃力。在本研究中，我们提出通过引入对话模型GAEA来解决这一问题，该模型可根据用户需求提供图像的位置相关信息。目前不存在能够支撑这类模型训练的大规模数据集，因此我们提出了GAEA-1.4M这一综合性数据集，它包含超过80万张图像和约140万对问答对，是利用OpenStreetMap（OSM，开放街图）属性和地理上下文线索构建的。为进行定量评估，我们提出了多样化基准GAEA-Bench，它包含3500对图文对，用于评估具备多样问题类型的对话能力。我们测试了11个最先进的开源及专有LMMs，结果表明GAEA的性能显著优于最优开源模型LLaVA-OneVision达18.2%，优于最优专有模型GPT-4o达7.2%。我们的数据集、模型和代码均已公开。
英文摘要：
Image geolocalization, in which an AI model traditionally predicts the precise GPS coordinates of an image, is a challenging task with many downstream applications. However, the user cannot utilize the model to further their knowledge beyond the GPS coordinates; the model lacks an understanding of the location and the conversational ability to communicate with the user. In recent days, with the tremendous progress of large multimodal models (LMMs) -- proprietary and open-source -- researchers have attempted to geolocalize images via LMMs. However, the issues remain unaddressed; beyond general tasks, for more specialized downstream tasks, such as geolocalization, LMMs struggle. In this work, we propose solving this problem by introducing a conversational model, GAEA, that provides information regarding the location of an image as the user requires. No large-scale dataset enabling the training of such a model exists. Thus, we propose GAEA-1.4M, a comprehensive dataset comprising over 800k images and approximately 1.4M question-answer pairs, constructed by leveraging OpenStreetMap (OSM) attributes and geographical context clues. For quantitative evaluation, we propose a diverse benchmark, GAEA-Bench, comprising 3.5k image-text pairs to evaluate conversational capabilities equipped with diverse question types. We consider 11 state-of-the-art open-source and proprietary LMMs and demonstrate that GAEA significantly outperforms the best open-source model, LLaVA-OneVision, by 18.2% and the best proprietary model, GPT-4o, by 7.2%. Our dataset, model and codes are available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
