从实验室到现实应用：评估边缘设备上用于移动机器人的零样本场景解读
From the Laboratory to Real-World Application: Evaluating Zero-Shot Scene Interpretation on Edge Devices for Mobile Robotics
AI总结：
本文评估了适用于移动机器人边缘设备的小型视觉语言模型在零样本场景解读任务中的性能与应用潜力。
摘要：
视频理解、场景解读和常识推理是高度具有挑战性的任务，使代理能够解读视觉信息，从而使其能够感知、与环境互动并做出理性决策。近年来，大型语言模型（LLMs）和视觉语言模型（VLMs）在这些领域取得了显著进展，使领域特定应用以及零样本开放词汇任务成为可能，结合了多个领域。然而，所需计算复杂性对在边缘设备和移动机器人上下文中应用它们构成了挑战，尤其是在准确性和推理时间之间的权衡。在本文中，我们研究了最先进的VLMs在场景解读和动作识别任务中的能力，特别关注能够部署到移动机器人上下文中的小型VLMs。所提出的流程在包含各种现实城市景观、校园和室内场景的多样化数据集上进行评估。实验评估讨论了这些小型模型在边缘设备上的潜力，特别强调了挑战、弱点、固有模型偏差以及所获得信息的应用。补充材料可通过以下仓库提供：https://datahub.rz.rptu.de/hstr-csrl-public/publications/scene-interpretation-on-edge-devices/
英文摘要：
Video Understanding, Scene Interpretation and Commonsense Reasoning are highly challenging tasks enabling the interpretation of visual information, allowing agents to perceive, interact with and make rational decisions in its environment. Large Language Models (LLMs) and Visual Language Models (VLMs) have shown remarkable advancements in these areas in recent years, enabling domain-specific applications as well as zero-shot open vocabulary tasks, combining multiple domains. However, the required computational complexity poses challenges for their application on edge devices and in the context of Mobile Robotics, especially considering the trade-off between accuracy and inference time. In this paper, we investigate the capabilities of state-of-the-art VLMs for the task of Scene Interpretation and Action Recognition, with special regard to small VLMs capable of being deployed to edge devices in the context of Mobile Robotics. The proposed pipeline is evaluated on a diverse dataset consisting of various real-world cityscape, on-campus and indoor scenarios. The experimental evaluation discusses the potential of these small models on edge devices, with particular emphasis on challenges, weaknesses, inherent model biases and the application of the gained information. Supplementary material is provided via the following repository: https://datahub.rz.rptu.de/hstr-csrl-public/publications/scene-interpretation-on-edge-devices/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
