arXiv 2606.276462026-07-09cs.CVphysics.optics

面向冻结视觉语言模型的VLM感知元光学前端设计

VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models

Department of Artificial Intelligence, Hanyang University（汉阳大学人工智能系）；School of Computational Science and Engineering, Georgia Institute of Technology（佐治亚理工学院计算科学与工程学院）；Department of Electronic Engineering, Hanyang University（汉阳大学电子工程系）

Chanik Kang, Raphaël Pestourie, Haejun Chung

备注：: 18 pages, 6 figures, 3 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出CODA框架，通过可微成像和伴随梯度更新优化连续密度元光学前端，直接优化冻结CLIP分类器的交叉熵损失，在ImageNet-100上将零样本准确率从53.75%提升至65.41%，并跨模型和数据集泛化。

摘要：

传统的机器视觉流水线通常依赖于产生清晰、可解释图像的高质量光学器件，因此光学设计一直由分辨率、像差校正和像素保真度等图像级标准驱动。然而，在尺寸、成本或形状因数受限的应用中，此类光学器件往往不切实际，而紧凑型元光学器件提供了一种有吸引力的替代方案，但在严格的物理效率限制下运行。我们提出了CODA，一个协同设计框架，该框架利用可微成像和基于麦克斯韦仿真的伴随梯度更新，为冻结模型识别优化连续密度元光学前端。CODA直接优化固定零样本CLIP分类器的交叉熵损失，无需学习重建、图像信号处理或图像保真度辅助目标。在ImageNet-100上的二维模拟成像基准测试中，CODA将CLIP ViT-L/14的零样本准确率从使用焦点集中基线的53.75±3.57%提高到65.41±3.99%。优化后的光学器件无需重新优化即可跨CLIP、SigLIP和DINOv2在ImageNet-100、CIFAR-100和Food-101上迁移。这些结果表明，在受限的元光学成像下，通过将光学设计与冻结视觉模型目标对齐而非传统图像形成标准，可以改善下游识别性能。

英文摘要：

Conventional machine-vision pipelines typically rely on high-quality optics that produce clean, human-interpretable images, and optical design has therefore been driven by image-level criteria such as resolution, aberration correction, and pixel fidelity. However, such optics are often impractical for size-, cost-, or form-factor-constrained applications, where compact meta-optics offer an attractive alternative but operate under strict physical efficiency limits. We propose CODA, a co-design framework that optimizes a continuous-density meta-optic front-end for frozen-model recognition using differentiable image formation and adjoint-gradient updates of Maxwell-based simulations. CODA directly optimizes the cross-entropy loss of a fixed zero-shot CLIP classifier without learned reconstruction, image signal processing, or image-fidelity auxiliary objectives. In a two-dimensional simulated imaging benchmark on ImageNet-100, CODA improves CLIP ViT-L/14 zero-shot accuracy from 53.75 $\pm$ 3.57$\%$ with a focal-concentration baseline to 65.41 $\pm$ 3.99$\%$. The optimized optics further transfer without re-optimization across CLIP, SigLIP, and DINOv2 on ImageNet-100, CIFAR-100, and Food-101. These results demonstrate that, under constrained meta-optic imaging, downstream recognition can be improved by aligning optical design with frozen vision-model objectives rather than conventional image-formation criteria.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。