FlexiFly:借助可重构无人机系统赋能基础模型与物理世界交互
FlexiFly: Interfacing the Physical World with Foundation Models Empowered by Reconfigurable Drone Systems
AI总结:
提出FlexiFly平台,通过图像分割技术和可重构无人机系统,使基础模型能“放大”分析物理环境相关区域,在智能家居任务中成功率提升达85%,推动基础模型与物理世界自然交互。
摘要:
基础模型(FM)在生成数字媒体方面展现出巨大的人类般能力。然而,能够自由感知、交互和驱动物理领域的基础模型还远未实现。这是因为1)需要密集部署传感器以完全覆盖和分析大空间,而2)事件通常局限于小区域,使得FM难以准确定位与当前任务相关的感兴趣区域。我们提出了FlexiFly,这是一个使FM能够“放大”并以更高粒度分析相关区域的平台,以更好地理解物理环境并执行任务。FlexiFly通过引入1)一种新颖的图像分割技术,帮助识别相关位置,以及2)一个模块化且可重构的传感与驱动无人机平台,使FM能够利用相关传感器和执行器进行“放大”来实现这一目标。我们通过真实的智能家居部署证明,FlexiFly使FM和LLM能够以高达85%的成功率完成各种任务。FlexiFly是朝着能够自然与物理世界交互的FM和LLM迈出的关键一步。
英文摘要:
Foundation models (FM) have shown immense human-like capabilities for generating digital media. However, foundation models that can freely sense, interact, and actuate the physical domain is far from being realized. This is due to 1) requiring dense deployments of sensors to fully cover and analyze large spaces, while 2) events often being localized to small areas, making it difficult for FMs to pinpoint relevant areas of interest relevant to the current task. We propose FlexiFly, a platform that enables FMs to ``zoom in'' and analyze relevant areas with higher granularity to better understand the physical environment and carry out tasks. FlexiFly accomplishes by introducing 1) a novel image segmentation technique that aids in identifying relevant locations and 2) a modular and reconfigurable sensing and actuation drone platform that FMs can actuate to ``zoom in'' with relevant sensors and actuators. We demonstrate through real smart home deployments that FlexiFly enables FMs and LLMs to complete diverse tasks up to $85\%$ more successfully. FlexiFly is critical step towards FMs and LLMs that can naturally interface with the physical world.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。