VSD2M:用于多帧动画贴纸生成的大规模视觉语言贴纸数据集
VSD2M: A Large-scale Vision-language Sticker Dataset for Multi-frame Animated Sticker Generation
AI总结:
本文提出VSD2M,一个两百万规模的最大视觉语言贴纸数据集,并设计时空交互层以提升动画贴纸生成性能,建立全面基准。
摘要:
作为社交媒体中一种常见的交流形式,贴纸因其能够以生动、可爱且有趣的方式传达情感而在互联网场景中赢得用户的喜爱。人们倾向于通过检索而非创作来获取合适的贴纸,因为创作贴纸耗时且依赖于功能有限、基于规则的创意工具。如今,先进的文本到视频算法催生了众多通用视频生成系统,这些系统允许用户仅通过提供简单的文本提示即可定制高质量、照片级逼真的视频。然而,创建定制动画贴纸(其帧率低于视频且语义更抽象)在很大程度上受到数据获取困难和基准不完整的阻碍。为了促进研究人员在动画贴纸生成(ASG)领域的探索,我们首先构建了目前最大的视觉语言贴纸数据集,名为VSD2M,规模达两百万级,包含静态和动画贴纸。其次,为了提升传统视频生成方法在具有离散特征的ASG任务上的性能,我们提出了一种时空交互(STI)层,利用语义交互和细节保留来解决信息利用不足的问题。此外,我们在VSD2M上训练了多种视频生成方法(如基于Transformer的方法、基于扩散的方法)的基线模型,并进行了详细分析,以建立对ASG任务的系统性监督。据我们所知,这是用于多帧动画贴纸生成的最全面的大规模基准,我们希望这项工作能为其他学者在智能创作方面提供有价值的启发。
英文摘要:
As a common form of communication in social media,stickers win users' love in the internet scenarios, for their ability to convey emotions in a vivid, cute, and interesting way. People prefer to get an appropriate sticker through retrieval rather than creation for the reason that creating a sticker is time-consuming and relies on rule-based creative tools with limited capabilities. Nowadays, advanced text-to-video algorithms have spawned numerous general video generation systems that allow users to customize high-quality, photo-realistic videos by only providing simple text prompts. However, creating customized animated stickers, which have lower frame rates and more abstract semantics than videos, is greatly hindered by difficulties in data acquisition and incomplete benchmarks. To facilitate the exploration of researchers in animated sticker generation (ASG) field, we firstly construct the currently largest vision-language sticker dataset named VSD2M at a two-million scale that contains static and animated stickers. Secondly, to improve the performance of traditional video generation methods on ASG tasks with discrete characteristics, we propose a Spatial Temporal Interaction (STI) layer that utilizes semantic interaction and detail preservation to address the issue of insufficient information utilization. Moreover, we train baselines with several video generation methods (e.g., transformer-based, diffusion-based methods) on VSD2M and conduct a detailed analysis to establish systemic supervision on ASG task. To the best of our knowledge, this is the most comprehensive large-scale benchmark for multi-frame animated sticker generation, and we hope this work can provide valuable inspiration for other scholars in intelligent creation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。