通过双通道文本提示和图像增强的CLIP实现视觉词义消歧
Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations
AI总结：
本文提出通过CLIP和双通道提示与图像增强实现视觉词义消歧，提升MRR和命中率，验证了精确提示的有效性。
摘要：
词义歧义在大语言模型（LLMs）的自然语言理解中始终是一个挑战。为了更好地理解如何通过视觉领域解决词汇歧义，我们开发了一个可解释的视觉词义消歧（VWSD）框架。该模型利用CLIP将歧义语言和候选图像投影到共享的多模态空间中。我们通过双重通道的语义和基于照片的提示集合来丰富文本嵌入，同时通过稳健的测试时增强来优化图像嵌入。然后使用余弦相似度确定与歧义文本最匹配的图像。在SemEval-2023 VWSD数据集上评估时，丰富嵌入将MRR从0.7227提升到0.7590，将命中率从0.5810提升到0.6220。消融研究显示，双通道提示提供强而低延迟的性能，而激进的图像增强仅带来微小收益。此外，使用WordNet定义和多语言提示集合的额外实验进一步表明，嘈杂的外部信号倾向于稀释语义特异性，强化了精确的CLIP对齐提示在视觉词义消歧中的有效性。
英文摘要：
Ambiguity poses persistent challenges in natural language understanding for large language models (LLMs). To better understand how lexical ambiguity can be resolved through the visual domain, we develop an interpretable Visual Word Sense Disambiguation (VWSD) framework. The model leverages CLIP to project ambiguous language and candidate images into a shared multimodal space. We enrich textual embeddings using a dual-channel ensemble of semantic and photo-based prompts with WordNet synonyms, while image embeddings are refined through robust test-time augmentations. We then use cosine similarity to determine the image that best aligns with the ambiguous text. When evaluated on the SemEval-2023 VWSD dataset, enriching the embeddings raises the MRR from 0.7227 to 0.7590 and the Hit Rate from 0.5810 to 0.6220. Ablation studies reveal that dual-channel prompting provides strong, low-latency performance, whereas aggressive image augmentation yields only marginal gains. Additional experiments with WordNet definitions and multilingual prompt ensembles further suggest that noisy external signals tend to dilute semantic specificity, reinforcing the effectiveness of precise, CLIP-aligned prompts for visual word sense disambiguation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
