XMeCap：具有子图像适应性的模因说明文字生成
XMeCap: Meme Caption Generation with Sub-Image Adaptability
AI总结：
针对多图像模因说明文字生成问题，提出结合监督微调、强化学习及全局/局部图文相似性奖励的 XMeCap 框架，显著提升单图与多图模因生成效果。
摘要：
幽默深植于社会意义与文化细节之中，对机器构成了独特挑战。尽管自然语言处理已取得进展，但现实世界中的幽默往往存在于多模态语境中，并以模因（meme）为典型载体。本文特别强调多图像对模因说明文字生成的影响。随后，我们提出 XMeCap 框架，这是一种采用监督微调与强化学习的新方法，并基于一个创新奖励模型，该模型同时考虑视觉与文本之间的全局和局部相似性。我们以当代模型为基准进行评估，结果表明，该方法在单图像和多图像模因以及不同模因类别的说明文字生成上均有显著提升。XMeCap 在单图像模因上取得 75.85 的平均评估分数，在多图像模因上取得 66.32 的平均评估分数，分别比最佳基线高出 6.75% 和 8.56%。本研究不仅为模因相关研究建立了新前沿，也凸显了机器在多模态环境中理解并生成幽默的潜力。
英文摘要：
Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines. While advances have been made in natural language processing, real-world humor often thrives in a multi-modal context, encapsulated distinctively by memes. This paper poses a particular emphasis on the impact of multi-images on meme captioning. After that, we introduce the \textsc{XMeCap} framework, a novel approach that adopts supervised fine-tuning and reinforcement learning based on an innovative reward model, which factors in both global and local similarities between visuals and text. Our results, benchmarked against contemporary models, manifest a marked improvement in caption generation for both single-image and multi-image memes, as well as different meme categories. \textsc{XMeCap} achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 6.75\% and 8.56\%, respectively. This research not only establishes a new frontier in meme-related studies but also underscores the potential of machines in understanding and generating humor in a multi-modal setting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
