面向遥感文本到图像检索的神经符号推理：为复杂查询的基座模型
Neurosymbolic Inference On Foundation Models For Remote Sensing Text-to-image Retrieval With Complex Queries
AI总结：
RUNE通过结合大型语言模型和神经符号AI，利用逻辑推理提升遥感文本到图像检索的性能与可解释性，针对复杂查询和图像不确定性提出新的评估指标。
摘要：
遥感（RS）领域的文本到图像检索随着大型视觉-语言模型（LVLMs）的兴起而迅速发展，这些模型专门针对航空和卫星图像进行优化，最终形成了遥感大型视觉-语言模型（RS-LVLMS）。然而，有限的可解释性和对复杂空间关系的处理能力仍然是实际应用中的关键挑战。为了解决这些问题，我们引入了RUNE（使用神经符号实体进行推理）方法，该方法结合了大型语言模型（LLMs）和神经符号AI，通过推理检测到的实体与从文本查询中导出的谓词逻辑（FOL）表达式之间的兼容性来检索图像。与依赖隐式联合嵌入的RS-LVLMs不同，RUNE执行显式推理，提高了性能和可解释性。为了可扩展性，我们提出了一种逻辑分解策略，该策略在条件子集上操作检测到的实体，保证了执行时间比神经方法更短。而不是使用基座模型进行端到端检索，我们仅利用它们生成FOL表达式，将推理委托给神经符号推理模块。在评估中，我们重新利用DOTA数据集，该数据集最初是为对象检测设计的，通过增加比现有基准更复杂的查询来增强它。我们展示了LLM在文本到逻辑翻译中的有效性，并将RUNE与最先进的RS-LVLMs进行比较，证明了其优越的性能。我们引入了两个指标，检索鲁棒性到查询复杂性（RRQC）和检索鲁棒性到图像不确定性（RRIU），这些指标评估性能相对于查询复杂性和图像不确定性。RUNE在复杂的RS检索任务中优于联合嵌入模型，提供了性能、鲁棒性和可解释性的提升。我们通过洪水后卫星图像检索的用例展示了RUNE在实际RS应用中的潜力。
英文摘要：
Text-to-image retrieval in remote sensing (RS) has advanced rapidly with the rise of large vision-language models (LVLMs) tailored for aerial and satellite imagery, culminating in remote sensing large vision-language models (RS-LVLMS). However, limited explainability and poor handling of complex spatial relations remain key challenges for real-world use. To address these issues, we introduce RUNE (Reasoning Using Neurosymbolic Entities), an approach that combines Large Language Models (LLMs) with neurosymbolic AI to retrieve images by reasoning over the compatibility between detected entities and First-Order Logic (FOL) expressions derived from text queries. Unlike RS-LVLMs that rely on implicit joint embeddings, RUNE performs explicit reasoning, enhancing performance and interpretability. For scalability, we propose a logic decomposition strategy that operates on conditioned subsets of detected entities, guaranteeing shorter execution time compared to neural approaches. Rather than using foundation models for end-to-end retrieval, we leverage them only to generate FOL expressions, delegating reasoning to a neurosymbolic inference module. For evaluation we repurpose the DOTA dataset, originally designed for object detection, by augmenting it with more complex queries than in existing benchmarks. We show the LLM's effectiveness in text-to-logic translation and compare RUNE with state-of-the-art RS-LVLMs, demonstrating superior performance. We introduce two metrics, Retrieval Robustness to Query Complexity (RRQC) and Retrieval Robustness to Image Uncertainty (RRIU), which evaluate performance relative to query complexity and image uncertainty. RUNE outperforms joint-embedding models in complex RS retrieval tasks, offering gains in performance, robustness, and explainability. We show RUNE's potential for real-world RS applications through a use case on post-flood satellite image retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
