利用YOLO-World与GPT-4V大语言多模态模型实现无人机影像中的零样本行人检测与动作识别
Leveraging YOLO-World and GPT-4V LMMs for Zero-Shot Person Detection and Action Recognition in Drone Imagery
AI总结：
本文研究零样本大型多模态模型在无人机感知中的应用，评估YOLO-World和GPT-4V在航拍影像中的行人检测与动作识别表现，发现YOLO-World检测性能良好，而GPT-4V在动作分类上较弱但擅长过滤区域提议和场景描述，为后续研究奠定基础。
摘要：
本文探索了零样本大型多模态模型（LMMs）在无人机感知领域的潜力。我们聚焦于行人检测与动作识别任务，并使用一个公开的航拍视角数据集评估了两个具有代表性的LMMs，即YOLO-World和GPT-4V(ision)。传统深度学习方法高度依赖大规模且高质量的训练数据集。然而，在某些机器人应用场景中，获取此类数据集可能资源消耗巨大，或难以在合理时间内完成。基于提示的大型多模态模型（LMMs）的灵活性及其卓越的泛化能力，有望在这些场景中彻底变革机器人应用。我们的研究结果表明，YOLO-World展现出良好的检测性能。GPT-4V在准确分类动作类别方面存在困难，但在过滤不想要的区域提议以及提供场景整体描述方面取得了有前景的结果。本研究是利用LMMs进行无人机感知的初步尝试，并为该领域未来的研究奠定了基础。
英文摘要：
In this article, we explore the potential of zero-shot Large Multimodal Models (LMMs) in the domain of drone perception. We focus on person detection and action recognition tasks and evaluate two prominent LMMs, namely YOLO-World and GPT-4V(ision) using a publicly available dataset captured from aerial views. Traditional deep learning approaches rely heavily on large and high-quality training datasets. However, in certain robotic settings, acquiring such datasets can be resource-intensive or impractical within a reasonable timeframe. The flexibility of prompt-based Large Multimodal Models (LMMs) and their exceptional generalization capabilities have the potential to revolutionize robotics applications in these scenarios. Our findings suggest that YOLO-World demonstrates good detection performance. GPT-4V struggles with accurately classifying action classes but delivers promising results in filtering out unwanted region proposals and in providing a general description of the scenery. This research represents an initial step in leveraging LMMs for drone perception and establishes a foundation for future investigations in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
