EdgeSAM：提示在环中的SAM蒸馏
EdgeSAM: Prompt-In-the-Loop Distillation for SAM
AI总结：
EdgeSAM通过将SAM的ViT编码器蒸馏为CNN，并在蒸馏循环中引入提示编码器和掩码解码器，实现了边缘设备上37倍加速，同时提升COCO和LVIS的mIoU，并支持iPhone 14上30 FPS运行。
摘要：
本文介绍了EdgeSAM，它是Segment Anything Model（SAM）的一种加速变体，专为在边缘设备上高效执行而优化，同时将性能损失降至最低。我们的方法是将基于ViT的原始SAM图像编码器蒸馏为纯CNN架构，该架构更适合边缘设备。我们仔细评估了各种蒸馏策略，并证明与任务无关的编码器蒸馏无法完全捕捉SAM所包含的全部知识。为克服这一瓶颈，我们将提示编码器和掩码解码器纳入蒸馏过程，并在循环中加入框和点提示，使蒸馏模型能够准确捕捉用户输入与掩码生成之间的复杂动态。为缓解点提示蒸馏带来的数据集偏差问题，我们在编码器内加入了一个轻量级模块。最终，EdgeSAM相比原始SAM实现了37倍的速度提升，并且在边缘设备上部署时，其速度是MobileSAM/EfficientSAM的7倍以上，同时在COCO和LVIS上的mIoU分别提高了2.3/1.5和3.1/1.6。它也是首个能在iPhone 14上以超过30 FPS运行的SAM变体。代码和演示可在https://www.mmlab-ntu.com/project/edgesam获取。
英文摘要：
This paper presents EdgeSAM, an accelerated variant of the Segment Anything Model (SAM), optimized for efficient execution on edge devices with minimal compromise in performance. Our approach involves distilling the original ViT-based SAM image encoder into a purely CNN-based architecture, better suited for edge devices. We carefully benchmark various distillation strategies and demonstrate that task-agnostic encoder distillation fails to capture the full knowledge embodied in SAM. To overcome this bottleneck, we include both the prompt encoder and mask decoder in the distillation process, with box and point prompts in the loop, so that the distilled model can accurately capture the intricate dynamics between user input and mask generation. To mitigate dataset bias issues stemming from point prompt distillation, we incorporate a lightweight module within the encoder. As a result, EdgeSAM achieves a 37-fold speed increase compared to the original SAM, and it also outperforms MobileSAM/EfficientSAM, being over 7 times as fast when deployed on edge devices while enhancing the mIoUs on COCO and LVIS by 2.3/1.5 and 3.1/1.6, respectively. It is also the first SAM variant that can run at over 30 FPS on an iPhone 14. Code and demo are available at https://www.mmlab-ntu.com/project/edgesam.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
