TinyVLM：通过Matryoshka嵌入的视觉-语言蒸馏实现微控制器上的零样本目标检测
TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings
AI总结：
TinyVLM通过Matryoshka嵌入和视觉-语言蒸馏，在微控制器上实现低内存的零样本目标检测
摘要：
零样本目标检测能够识别新对象而无需特定任务训练，但当前方法依赖于大型视觉语言模型（如CLIP），需要数百兆字节的内存——远超微控制器单元（MCU）的限制。我们提出了TinyVLM，这是首个能够在资源受限的MCU上实现零样本目标检测的框架，其内存占用低于1MB。我们的方法引入了三个关键创新：（1）解耦架构，将视觉推理与文本编码分离，允许预计算的类别嵌入存储在闪存中；（2）Matryoshka蒸馏，训练多维（16-256）嵌套嵌入，实现灵活的精度-内存权衡；（3）量化嵌入存储，将类别原型内存减少4倍，同时最小化精度损失。在Conceptual Captions 3M（CC3M）上训练后，TinyVLM在COCO、Flowers102和Food101上实现了具有竞争力的零样本精度，仅需285KB的RAM和892KB的闪存内存用于部署的视觉编码器。我们在STM32H7上实现了26 FPS的实时推理，在MAX78000上利用其CNN加速器实现了超过1000 FPS的推理速度，首次在边缘设备上实现了实用的零样本检测。
英文摘要：
Zero-shot object detection enables recognising novel objects without task-specific training, but current approaches rely on large vision language models (VLMs) like CLIP that require hundreds of megabytes of memory - far exceeding the constraints of micro controller units (MCUs). We present TinyVLM, the first framework enabling zero-shot object detection on resource-constrained MCUs with less than 1MB of memory. Our approach introduces three key innovations: (1) a decoupled architecture that separates visual inference from text encoding, allowing precomputed class embeddings to be stored in flash memory; (2) Matryoshka distillation that trains nested embeddings at multiple dimensions (16-256), enabling flexible accuracy-memory trade-offs; and (3) quantized embedding storage that reduces class prototype memory by 4x with minimal accuracy loss. Trained on Conceptual Captions 3M (CC3M), TinyVLM achieves competitive zero-shot accuracy on COCO, Flowers102, and Food101 while requiring only 285KB of RAM and 892KB of flash memory for the deployed vision encoder. We demonstrate real-time inference at 26 FPS on STM32H7 and over 1,000 FPS on MAX78000 with its CNN accelerator, enabling practical zero-shot detection on edge devices for the first time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
