UAVBench 和 UAVIT-1M：用于低空无人机视觉-语言理解的LLM基准测试与增强
UAVBench and UAVIT-1M: Benchmarking and Enhancing MLLMs for Low-Altitude UAV Vision-Language Understanding
AI总结：
本文提出UAVBench和UAVIT-1M，用于评估和提升LLM在低空无人机视觉-语言任务中的能力，通过大规模数据集和基准测试揭示开源LLM在低空视觉内容生成上的不足，并通过微调提升性能。
摘要：
多模态大语言模型（MLLMs）在自然图像和卫星遥感图像上取得了显著进展。然而，理解低空无人机场景仍是一个挑战。现有数据集主要集中在少数特定的低空视觉任务上，无法全面评估MLLMs在真实低空无人机应用中的能力。因此，我们引入UAVBench，一个全面的基准，和UAVIT-1M，一个大规模的指令微调数据集，用于评估和提升MLLMs在低空视觉-语言任务中的能力。UAVBench包含43个测试单元和966k高质量数据样本，覆盖10个任务在图像级和区域级。UAVIT-1M包含约124万条多样化的指令，涵盖789k多场景图像和约2000种空间分辨率，共11个不同任务。UAVBench和UAVIT-1M包含纯现实世界视觉图像和丰富的天气条件，并涉及人工验证以确保高质量。我们对11种最先进的MLLMs进行深入分析，发现开源MLLMs无法生成准确的低空视觉内容对话，落后于闭源MLLMs。大量实验表明，对UAVIT-1M进行微调可以显著缩小这一差距。我们的贡献为弥合当前MLLMs与低空无人机真实应用场景需求之间的差距铺平了道路。（项目页面：https://UAVBench.github.io/）
英文摘要：
Multimodal Large Language Models (MLLMs) have made significant strides in natural images and satellite remote sensing images. However, understanding low-altitude drone scenarios remains a challenge. Existing datasets primarily focus on a few specific low-altitude visual tasks, which cannot fully assess the ability of MLLMs in real-world low-altitude UAV applications. Therefore, we introduce UAVBench, a comprehensive benchmark, and UAVIT-1M, a large-scale instruction tuning dataset, designed to evaluate and improve MLLMs' abilities in low-altitude vision-language tasks. UAVBench comprises 43 test units and 966k high-quality data samples across 10 tasks at the image-level and region-level. UAVIT-1M consists of approximately 1.24 million diverse instructions, covering 789k multi-scene images and about 2,000 types of spatial resolutions with 11 distinct tasks. UAVBench and UAVIT-1M feature pure real-world visual images and rich weather conditions, and involve manual verification to ensure high quality. Our in-depth analysis of 11 state-of-the-art MLLMs using UAVBench reveals that open-source MLLMs cannot generate accurate conversations about low-altitude visual content, lagging behind closed-source MLLMs. Extensive experiments demonstrate that fine-tuning open-source MLLMs on UAVIT-1M significantly addresses this gap. Our contributions pave the way for bridging the gap between current MLLMs and low-altitude UAV real-world application demands. (Project page: https://UAVBench.github.io/)
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
