OSPC：以大语言模型为催化剂检测有害表情包
OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst
AI总结：
本文提出融合 BLIP、PP-OCR、TrOCR 与 Qwen 的多语言有害表情包检测框架，并通过 GPT-4V 标注数据蒸馏提升性能，在 Online Safety Prize Challenge 中以 AUROC 0.7749 排名第一。
摘要：
表情包在互联网上快速传播个人观点和立场，同时也在传播社会偏见与成见方面带来重大挑战。本研究提出了一种检测有害表情包的新方法，尤其针对新加坡多元文化、多语言的语境。该方法整合图像描述生成、光学字符识别（OCR）和大语言模型（LLM）分析，以全面理解并分类有害表情包。系统利用 BLIP 模型生成图像描述，使用 PP-OCR 和 TrOCR 进行多语言文本识别，并采用 Qwen LLM 实现细致的语言理解，因此能够识别以英语、中文、马来语和泰米尔语制作的表情包中的有害内容。为提升系统性能，作者利用借助 GPT-4V 标注的额外数据对方法进行微调，旨在将 GPT-4V 对有害表情包的理解能力蒸馏到本系统中。该框架在 AI Singapore 主办的 Online Safety Prize Challenge 公共排行榜上取得第一名，AUROC 为 0.7749，准确率为 0.7087，显著领先其他队伍。值得注意的是，该方法优于以往基准：FLAVA 的 AUROC 为 0.5695，VisualBERT 的 AUROC 为 0.5561。
英文摘要：
Memes, which rapidly disseminate personal opinions and positions across the internet, also pose significant challenges in propagating social bias and prejudice. This study presents a novel approach to detecting harmful memes, particularly within the multicultural and multilingual context of Singapore. Our methodology integrates image captioning, Optical Character Recognition (OCR), and Large Language Model (LLM) analysis to comprehensively understand and classify harmful memes. Utilizing the BLIP model for image captioning, PP-OCR and TrOCR for text recognition across multiple languages, and the Qwen LLM for nuanced language understanding, our system is capable of identifying harmful content in memes created in English, Chinese, Malay, and Tamil. To enhance the system's performance, we fine-tuned our approach by leveraging additional data labeled using GPT-4V, aiming to distill the understanding capability of GPT-4V for harmful memes to our system. Our framework achieves top-1 at the public leaderboard of the Online Safety Prize Challenge hosted by AI Singapore, with the AUROC as 0.7749 and accuracy as 0.7087, significantly ahead of the other teams. Notably, our approach outperforms previous benchmarks, with FLAVA achieving an AUROC of 0.5695 and VisualBERT an AUROC of 0.5561.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
