arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.10475cs.CVcs.MM
收藏

CBVS:面向真实短视频搜索场景的大规模中文图文基准

CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios

Tencent PCG(腾讯平台与内容事业群);University of Science and Technology of China(中国科学技术大学)

Xiangshuo Qiao, Xianxin Li, Xiaozhe Qu, Jie Zhang, Yang Liu, Yu Luo, Cihang Jin, Jin Ma

AI总结:

本文构建首个中文短视频搜索图文基准 CBVS,并提出训练时利用封面文字、推理时不依赖该模态的 UniCLIP,实验与线上部署均验证其显著效果。

摘要:

在大规模图文数据集上预训练的视觉-语言模型,已在图像检索等下游任务中表现出卓越性能。用于预训练的大多数图像以开放领域常识性视觉元素的形式呈现。与之不同,短视频搜索场景中的视频封面以用户原创内容的形式呈现,可提供视频的重要视觉摘要。此外,一部分视频封面带有人工设计的封面文字,可提供语义补充。为填补短视频封面数据方面的空白,我们建立了首个面向中文短视频搜索场景的大规模封面-文字基准。具体而言,我们发布了两个大规模数据集 CBVS-5M/10M 以提供短视频封面,并发布人工精细标注数据集 CBVS-20K 以提供真实用户查询,后者作为中文短视频搜索领域的图文基准测试。为在模态缺失情况下融合封面文字的语义,我们提出 UniCLIP:封面文字在训练期间发挥引导作用,但在推理时不被依赖。在 CBVS-20K 上的大量评估证明了我们所提方法的优异性能。UniCLIP 已部署到腾讯访问量达数亿次的在线视频搜索系统中,并取得显著收益。数据集和代码可在 https://github.com/QQBrowserVideoSearch/CBVS-UniCLIP 获取。

英文摘要:

Vision-Language Models pre-trained on large-scale image-text datasets have shown superior performance in downstream tasks such as image retrieval. Most of the images for pre-training are presented in the form of open domain common-sense visual elements. Differently, video covers in short video search scenarios are presented as user-originated contents that provide important visual summaries of videos. In addition, a portion of the video covers come with manually designed cover texts that provide semantic complements. In order to fill in the gaps in short video cover data, we establish the first large-scale cover-text benchmark for Chinese short video search scenarios. Specifically, we release two large-scale datasets CBVS-5M/10M to provide short video covers, and the manual fine-labeling dataset CBVS-20K to provide real user queries, which serves as an image-text benchmark test in the Chinese short video search field. To integrate the semantics of cover text in the case of modality missing, we propose UniCLIP where cover texts play a guiding role during training, however are not relied upon by inference. Extensive evaluation on CBVS-20K demonstrates the excellent performance of our proposal. UniCLIP has been deployed to Tencent's online video search systems with hundreds of millions of visits and achieved significant gains. The dataset and code are available at https://github.com/QQBrowserVideoSearch/CBVS-UniCLIP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑