Open-SAT: 一种基于LLM的查询嵌入细化方法用于卫星影像开放词汇物体检索
Open-SAT: LLM-Guided Query Embedding Refinement for Open-Vocabulary Object Retrieval in Satellite Imagery
AI总结：
Open-SAT通过LLM引导在推理阶段优化查询与卫星影像内容的对齐，提升开放词汇检索性能，实验表明其F1分数提升达16.04%。
摘要：
在卫星应用中，用户查询通常为开放式的自然语言，超出预定义类别。这给图像检索带来挑战，因为系统需泛化到未见过的对象和概念。尽管视觉-语言模型（VLMs）如CLIP被广泛用于文本-图像检索，但即使微调变体也难以准确对齐此类查询与卫星影像。为此，我们提出Open-SAT，一种无需训练的查询嵌入细化算法，通过推理阶段优化查询与卫星图像内容的对齐。Open-SAT利用VLMs计算图像块的嵌入，并存储在向量数据库中以实现高效检索。在查询阶段，它利用大型语言模型（LLMs）通过整合感兴趣对象及其周围环境的上下文信息来细化文本嵌入。阈值自由的检索机制进一步提升了准确性和效率。在三个公开基准测试中，实验结果表明Open-SAT将F1分数提升高达16.04%，同时检索的图像块数量相当。这些结果证明了Open-SAT在开放词汇卫星图像检索中的有效性，无需额外训练或监督。
英文摘要：
In satellite applications, user queries often take the form of open-ended natural language, extending beyond a fixed set of predefined categories. This open-vocabulary nature poses significant challenges for retrieving relevant image tiles, as the retrieval system must generalize to a wide range of unseen objects and concepts. While vision-language models (VLMs) such as CLIP are widely used for text-image retrieval, even fine-tuned variants often struggle to accurately align such queries with satellite imagery. To address this, we propose Open-SAT, a training-free query embedding refinement algorithm that operates at inference time to improve alignment between user queries and satellite image content. Open-SAT uses VLMs to compute embeddings for image tiles, which are stored in a vector database for efficient retrieval. At query time, it leverages Large Language Models (LLMs) to refine the text embeddings by incorporating contextual information about objects of interest and their surroundings. A threshold-free retrieval mechanism further enhances accuracy and efficiency. Experimental results in three public benchmarks demonstrate that Open-SAT improves the F1 score by up to 16.04%, while retrieving a comparable number of image tiles. These results demonstrate the effectiveness of Open-SAT in open-vocabulary satellite image retrieval, leveraging LLM guidance without the need for additional training or supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
