ClimateVID -- 社交媒体视频分析及涉及的挑战
ClimateVID -- Social Media Videos Analysis and Challenges Involved
AI总结：
本文通过零样本和聚类方法分析社交媒体视频，探讨视觉主题检测的挑战，发现ConvNeXt V2和DINOv2在气候议题聚类中表现各异。
摘要：
数字内容的迅猛增长，尤其是社交媒体上的短视频，显著改变了公共讨论中话题的讨论和理解方式。本文通过评估零样本和聚类能力，推进自动化视觉主题检测。我们评估了VideoChatGPT、PandaGPT和VideoLLava等显著VLMs的零样本图像分类能力，并将其与帧级CLIP图像分类基线进行比较。通过将聚类视为最小成本多割问题，我们旨在以无监督方式揭示有价值的模式。对于两种分析策略，我们提供了广泛的评估和实际指导。尽管VLMs目前无法检测特定的气候变化类别，但聚类结果是独特的视觉帧。鉴于VLMs目前无法把握气候变化讨论，我们专注于图像嵌入模型的聚类评估。我们发现ConvNeXt V2和DINOv2均产生有意义的聚类，其中DINOv2更关注风格差异和抽象类别，而ConvNeXt V2的聚类则在更细粒度上有所不同。代码可在https://github.com/KathPra/ClimateVID.git获取。
英文摘要：
The pervasive growth of digital content, specifically short videos on social media platforms, has significantly altered how topics are discussed and understood in public discourse. In this work, we advance automated visual theme detection by assessing zero-shot and clustering capabilities on social media data. (1) We evaluated the capabilities of notable VLMs such as VideoChatGPT, PandaGPT, and VideoLLava using zero-shot image classification and compared their performance to the baseline provided by frame-wise CLIP image classification. (2) By treating clustering as a minimum cost multicut problem, we aim to uncover insightful patterns in an unsupervised manner. For both analysis strategies, we provide extensive evaluations and practical guidance to practitioners. While VLMs are currently not able to detect climate change specific classes, the clustering results are distinct visual frames. %Given that VLMs are not currently capable to grasp the climate change discourse, we focus the clustering evaluation of image embedding models. We find that both ConvNeXt V2 and DINOv2 produce meaningful clusters, with DINOv2 focusing more on style differences and abstract categories, while ConvNeXt V2 clusters differ in more fine-grained ways. Code available at https://github.com/KathPra/ClimateVID.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
