arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.17499cs.LG
收藏

t-gems: 基于文本引导的退出模块用于减少clip图像编码器

t-gems: text-guided exit modules for decreasing clip image encoder

Samsung AI Center Warsaw(三星AI中心华沙);Institute of Fundamental Technological Research, PAS, Warsaw(基础技术研究所,波兰科学院,华沙)

Alberto Presta, Grzegorz Stefanski, Michal Byra, Krzysztof Arendt

AI总结:

本文提出t-gems文本引导退出模块,通过利用编码器中间层的语义内容分布,减少clip图像编码器的计算成本,同时保持跨模态理解性能。

摘要:

多模态深度神经网络通过整合多种数据模态来增强深度理解。不同模态的数据通常被投影到共享的潜在空间中进行相似性计算,但这一过程由于大型图像编码器和预测期间对测试数据的等量处理而变得资源密集。早期退出方法通过利用中间层来减少计算负载,节省时间和内存。然而,对于像图像-文本对这样的多模态数据开发此类方法具有挑战性。本研究探讨了编码器如clip中中间层中存在的语义内容分布,这些分布可以从文本描述中推导出来。我们引入了文本引导退出模块(t-gems)和基于速率的正则化器,以控制编码器的使用成本,同时保持跨模态理解性能。

英文摘要:

Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑