迈向准确的无人机图像感知：通过更强的任务提示引导视觉-语言模型
Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Prompts
AI总结：
AerialVP通过增强任务提示提升无人机图像感知性能，引入AerialSense基准评估模型在复杂场景下的表现。
摘要：
现有的基于VLMs的图像感知方法通常遵循一种范式，即模型根据用户提供的文本任务提示提取和分析图像内容。然而，当应用于无人机图像时，面临目标混淆、尺度变化和复杂背景等挑战。这些挑战出现的原因是VLMs对图像内容的理解依赖于视觉和文本标记之间的语义对齐。当任务提示简单而图像内容复杂时，实现有效的对齐变得困难，限制了模型对任务相关信息的关注能力。为了解决这个问题，我们引入了AerialVP，这是首个用于无人机图像感知中任务提示增强的代理框架。AerialVP主动从无人机图像中提取多维辅助信息以增强任务提示，克服了传统基于VLMs方法的局限性。具体而言，增强过程包括三个阶段：（1）分析任务提示以确定任务类型和增强需求，（2）从工具库中选择适当的工具，（3）根据分析和选定的工具生成增强的任务提示。为了评估AerialVP，我们引入了AerialSense，一个全面的无人机图像感知基准，包括Aerial Visual Reasoning、Aerial Visual Question Answering和Aerial Visual Grounding任务。AerialSense为评估模型泛化能力和在不同分辨率、光照条件以及城市和自然场景中的性能提供了标准化基础。实验结果表明，AerialVP显著增强了任务提示指导，导致在开源和专有VLMs中都实现了稳定且显著的性能提升。我们的工作将在https://github.com/lostwolves/AerialVP上发布。
英文摘要：
Existing image perception methods based on VLMs generally follow a paradigm wherein models extract and analyze image content based on user-provided textual task prompts. However, such methods face limitations when applied to UAV imagery, which presents challenges like target confusion, scale variations, and complex backgrounds. These challenges arise because VLMs' understanding of image content depends on the semantic alignment between visual and textual tokens. When the task prompt is simplistic and the image content is complex, achieving effective alignment becomes difficult, limiting the model's ability to focus on task-relevant information. To address this issue, we introduce AerialVP, the first agent framework for task prompt enhancement in UAV image perception. AerialVP proactively extracts multi-dimensional auxiliary information from UAV images to enhance task prompts, overcoming the limitations of traditional VLM-based approaches. Specifically, the enhancement process includes three stages: (1) analyzing the task prompt to identify the task type and enhancement needs, (2) selecting appropriate tools from the tool repository, and (3) generating enhanced task prompts based on the analysis and selected tools. To evaluate AerialVP, we introduce AerialSense, a comprehensive benchmark for UAV image perception that includes Aerial Visual Reasoning, Aerial Visual Question Answering, and Aerial Visual Grounding tasks. AerialSense provides a standardized basis for evaluating model generalization and performance across diverse resolutions, lighting conditions, and both urban and natural scenes. Experimental results demonstrate that AerialVP significantly enhances task prompt guidance, leading to stable and substantial performance improvements in both open-source and proprietary VLMs. Our work will be available at https://github.com/lostwolves/AerialVP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
