TEVI: 基于稀疏自编码器的文本条件视觉表示编辑以改进视觉-语言对齐
TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
AI总结：
提出TEVI框架，利用稀疏自编码器解耦图像嵌入，并通过文本条件掩码模块选择性重构嵌入，以改善CLIP等视觉-语言模型的图像-文本对齐，在多个检索基准上取得提升。
摘要：
视觉-语言模型（如CLIP）由于共享图像-文本嵌入空间，对多种任务非常有用。尽管如此，图像和文本嵌入往往对齐不佳，影响下游性能。最近的研究表明，这可以归因于信息不平衡：图像包含的信息比其标题描述的更多。在这项工作中，我们提出了TEVI，一个利用标题作为信号来决定从图像嵌入中保留哪些信息的框架。具体来说，我们使用稀疏自编码器来解耦图像嵌入，并训练一个掩码模块，根据给定的标题选择性重构嵌入。在具有合成标题的受控设置中，我们展示了TEVI在保留标题描述的属性同时丢弃其他属性方面的有效性。通过将TEVI应用于在自然图像上训练的CLIP模型，我们进一步在粗粒度短标题（MS COCO, Flickr）和细粒度长标题（IIW, DOCCI）基准上实现了改进的检索性能，在更丰富的标题上获得更强的增益，并在RoCOCO基准上提高了鲁棒性。
英文摘要：
Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space. Despite this, the image and text embeddings are often poorly aligned, affecting downstream performance. Recent work has hypothesized that this can be attributed to an information imbalance: images contain more information than their captions describe. In this work, we propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings. Specifically, we use sparse autoencoders to disentangle image embeddings and train a masking module to selectively reconstruct the embedding based on a given caption. In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others. We find that this extends to CLIP models trained on natural images, where TEVI learns to mask meaningfully and allows retrieval based on conditioning. Finally, we use TEVI to achieve improved retrieval performance across coarse-grained and fine-grained benchmarks. Code available at https://github.com/neuroexplicit-saar/TEVI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
