arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14477cs.CVcs.AIcs.DB
收藏

SafeSora:通过人类偏好数据集实现文本生成视频的安全对齐研究

SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset

Center for AI Safety and Governance, Institute for AI, Peking University(北京大学人工智能研究院人工智能安全与治理中心)

Josef Dai, Tianle Chen, Xuyao Wang, Ziran Yang, Taiye Chen, Jiaming Ji, Yaodong Yang

AI总结:

为降低大视觉模型的有害输出风险,研究团队构建了涵盖有用性、无害性两大维度及细分项的SafeSora人类偏好数据集,并验证了其在视频审核、模型对齐等场景的应用价值,为文本生成视频对齐研究提供基础支撑。

摘要:

为降低大视觉模型(LVMs)生成有害输出的风险,我们推出了SafeSora数据集,以推动文本生成视频任务与人类价值观对齐的研究。该数据集围绕文本生成视频任务中的人类偏好,从有用性和无害性两大核心维度展开构建。为捕捉深层人类偏好并便于众包标注人员开展结构化推理,我们将有用性细分为4个子维度,将无害性细分为12个子类别,以此作为试点标注的依据。SafeSora数据集包含14711条独特提示词、由4种不同LVM生成的57333个独特视频,以及51691对人工标注的偏好标注对。我们进一步通过多项应用验证了SafeSora数据集的实用性,包括训练文本-视频审核模型,以及通过微调提示增强模块或扩散模型实现LVM与人类偏好的对齐。这些应用凸显了该数据集可作为文本生成视频对齐研究(如人类偏好建模、对齐算法的开发与验证)基础的潜力。

英文摘要:

To mitigate the risk of harmful outputs from large vision models (LVMs), we introduce the SafeSora dataset to promote research on aligning text-to-video generation with human values. This dataset encompasses human preferences in text-to-video generation tasks along two primary dimensions: helpfulness and harmlessness. To capture in-depth human preferences and facilitate structured reasoning by crowdworkers, we subdivide helpfulness into 4 sub-dimensions and harmlessness into 12 sub-categories, serving as the basis for pilot annotations. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations labeled by humans. We further demonstrate the utility of the SafeSora dataset through several applications, including training the text-video moderation model and aligning LVMs with human preference by fine-tuning a prompt augmentation module or the diffusion model. These applications highlight its potential as the foundation for text-to-video alignment research, such as human preference modeling and the development and validation of alignment algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑