KAFA：通过视觉-语言模型的知识增强特征适配重新思考图像广告理解
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models
AI总结：
本文首次基于预训练视觉-语言模型研究图像广告理解，提出知识增强的多模态特征适配策略KAFA，以提升对复杂广告场景的理解能力。
摘要：
图像广告理解是一项具有广泛现实应用的关键任务。尽管由于涉及多样的非典型场景、现实世界实体以及对场景文本的推理而极具挑战性，但如何解读图像广告仍相对未被充分探索，尤其是在以卓越泛化能力和适配能力为特征的基础视觉-语言模型（VLMs）时代。本文中，我们首次通过预训练视觉-语言模型的视角对图像广告理解进行了实证研究。我们对将这些视觉-语言模型适配到图像广告理解中的实际挑战进行了基准测试并予以揭示。我们提出了一种简单的特征适配策略，以有效融合图像广告的多模态信息，并进一步利用现实世界实体的知识对其进行增强。我们希望我们的研究能引起更多对图像广告理解的关注，该任务与广告行业广泛相关。
英文摘要：
Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
