MobileSAM2：用于空间智能的轻量级图像分割模型
MobileSAM2: Lightweight Segment Anything for Spatial Intelligence
- Hong Kong Polytechnic University（香港理工大学）
- Nanyang Technological University（南洋理工大学）
- Xidian University（西安电子科技大学）
- SparcAI Inc.（SparcAI公司）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
研究旨在使SAM2更适用于移动设备，提出超图知识蒸馏方法HyperKD，由时间和粒度超图知识蒸馏构成，能有效建模转移知识。还推出MobileSAM2家族，经实验验证其在多基准测试及具身AI任务上有良好泛化性能。
AI中文摘要：
近期的大型视频基础模型SAM2能够对图像和视频中的任何事物进行分割，是各种应用的强大基础模型。然而，许多此类用例需要在手机和笔记本电脑等资源受限的设备上运行。在这项工作中，我们旨在通过将重量级的SAM2提炼为轻量级模型，使SAM2对移动设备更友好，便于在移动设备上对图像和视频中的任何事物进行分割。为此，我们提出了超图知识蒸馏（HyperKD），将超图的概念引入知识蒸馏，旨在有效地建模和转移SAM2的可泛化和全面的知识。HyperKD由时间超图知识蒸馏和粒度超图知识蒸馏组成，它们分别构建超图，以显式地建模和提取SAM2的可泛化时间知识和全面的多粒度知识，然后通过将其与构建的超图对齐，将这些知识提炼到轻量级学生模型中。此外，我们还提出了MobileSAM2，这是一个新的轻量级SAM2家族，通过在模型尺寸缩减过程中使用HyperKD搜索最佳模型架构，在效率和有效性之间取得平衡。广泛的实验在多个基准上验证了MobileSAM2，并在具身人工智能任务上显示出有希望的泛化性能。
英文摘要：
The recent large video foundation model, SAM2, enables segment anything in both images and videos, serving as a powerful base model for various applications. However, many of such use cases require to operate on resource-constrained devices like mobile phones and laptops. In this work, we aim to make SAM2 more mobile-friendly by distilling the heavyweight SAM2 into a lightweight model, facilitating segment anything in both images and videos on mobile devices. To this end, we propose Hypergraphical Knowledge Distill (HyperKD), which introduces the idea of hypergraph into knowledge distillation, aiming to effectively model and transfer SAM2's generalizable and comprehensive knowledge. HyperKD consists of Temporal HyperKD and Granularity HyperKD that construct hypergraphs to explicitly model and extract the generalizable temporal knowledge and the comprehensive multi-granularity knowledge from SAM2 respectively, which are then distilled into the lightweight student model by aligning it with the constructed hypergraphs. Besides, we present MobileSAM2, a new family of lightweight SAM2 that balances efficiency and effectiveness via searching the best model architectures with HyperKD during model size reduction. Extensive experiments validate MobileSAM2 across multiple benchmarks and show promising generalization performance on embodied AI tasks.
