融合可解释视觉概念的多模态阿拉伯语图像描述
Multimodal Arabic Captioning with Interpretable Visual Concept Integration
AI总结：
本研究提出可解释的阿拉伯语图像描述框架VLCAP，结合多语言CLIP编码器检索视觉概念与视觉语言模型生成描述，验证了不同编码器-解码器组合的效果，可生成文化适配、准确的阿拉伯语描述。
摘要：
我们提出VLCAP，一种阿拉伯语图像描述框架，将基于CLIP的视觉标签检索与多模态文本生成相结合。VLCAP并非仅依赖端到端描述生成，而是以可解释的阿拉伯语视觉概念为生成基础，这些概念通过三个多语言编码器mCLIP、AraCLIP和Jina V4提取，每个编码器都单独进行了标签检索效果评估。我们从训练描述文本构建混合词汇表，并补充约2.1万个从Visual Genome数据集翻译而来的通用领域标签，涵盖物体、属性和场景。排名前k的检索标签被转化为流畅的阿拉伯语提示词，与原始图像一同输入视觉语言模型。在第二阶段，我们测试了Qwen-VL和Gemini Pro Vision的描述生成效果，共形成六种编码器-解码器配置。结果显示，mCLIP + Gemini Pro Vision取得了最优的BLEU-1（5.34%）和余弦相似度（60.01%），而AraCLIP + Qwen-VL获得了最高的LLM评审得分（36.33%）。这种可解释的流水线能够生成符合文化语境、上下文准确的阿拉伯语图像描述。
英文摘要：
We present VLCAP, an Arabic image captioning framework that integrates CLIP-based visual label retrieval with multimodal text generation. Rather than relying solely on end-to-end captioning, VLCAP grounds generation in interpretable Arabic visual concepts extracted with three multilingual encoders, mCLIP, AraCLIP, and Jina V4, each evaluated separately for label retrieval. A hybrid vocabulary is built from training captions and enriched with about 21K general domain labels translated from the Visual Genome dataset, covering objects, attributes, and scenes. The top-k retrieved labels are transformed into fluent Arabic prompts and passed along with the original image to vision-language models. In the second stage, we tested Qwen-VL and Gemini Pro Vision for caption generation, resulting in six encoder-decoder configurations. The results show that mCLIP + Gemini Pro Vision achieved the best BLEU-1 (5.34%) and cosine similarity (60.01%), while AraCLIP + Qwen-VL obtained the highest LLM-judge score (36.33%). This interpretable pipeline enables culturally coherent and contextually accurate Arabic captions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
