Vision-Language Models pre-trained on large-scale image-text datasets have shown superior performance in downstream tasks such as image retrieval. Most of the images for pre-training are presented in the form of open domain common-sense visual elements. Differently, video covers in short video search scenarios are presented as user-originated contents that provide important visual summaries of videos. In addition, a portion of the video covers come with manually designed cover texts that provide semantic complements. In order to fill in the gaps in short video cover data, we establish the first large-scale cover-text benchmark for Chinese short video search scenarios. Specifically, we release two large-scale datasets CBVS-5M/10M to provide short video covers, and the manual fine-labeling dataset CBVS-20K to provide real user queries, which serves as an image-text benchmark test in the Chinese short video search field. To integrate the semantics of cover text in the case of modality missing, we propose UniCLIP where cover texts play a guiding role during training, however are not relied upon by inference. Extensive evaluation on CBVS-20K demonstrates the excellent performance of our proposal. UniCLIP has been deployed to Tencent's online video search systems with hundreds of millions of visits and achieved significant gains. The dataset and code are available at https://github.com/QQBrowserVideoSearch/CBVS-UniCLIP.
Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training
University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团);Alibaba-NTU Singapore Joint Research Institute, Nanyang Technological University(阿里巴巴-南洋理工大学新加坡联合研究院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);Zhejiang University(浙江大学)
A Chinese Continuous Sign Language Dataset Based on Complex Environments
College of Intelligent Systems Science and Engineering(智能系统科学与工程学院);Harbin Engineering University(哈尔滨工程大学);Northwest Institute of Mechanical and Electrical Engineering(西北机电工程研究院);Kyungil University(庆北大学)
AISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);Beijing AISHELL Technology Co., Ltd.(北京AISHELL科技有限公司);AI Center, OPPO(OPPO人工智能中心)
FakingRecipe: Detecting Fake News on Short Video Platforms from the Perspective of Creative Process
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学)
Taking an Emotional Look at Video Paragraph Captioning
Tongji University(同济大学);Hong Kong Polytechnic University(香港理工大学)
将视觉数据转换为自然语言对于机器理解世界并与人类交互至关重要。在这项工作中,一个全面的研究进行视频段落字幕,目标是生成一个给定的视频段落级的描述。然而,目前的研究主要集中在检测客观事实,忽略了需要建立句子之间的逻辑关联,并发现更准确的情感相关的视频内容。这样的问题影响了预测字幕的流畅和丰富的表达,这远远低于人类的语言标准。为了解决这个问题,我们建议构建一个大规模的情感和逻辑驱动的多语言数据集。该数据集被命名为EMVPC(Emotional Video Paragraph Captioning),包含日常生活中广泛使用的53种情绪,与这些情绪相对应的376个常见场景,10,291个高质量视频和20,582个精心制作的中英文段落字幕。在这个新的数据集中还提供了相关的情感类别、场景标签、情感词标签和逻辑词标签。拟议的EMVPC数据集旨在提供全面的视频段落字幕,具有丰富的情感、连贯的逻辑和精致的表达,这也可以有益于视觉语言领域的其他任务。此外,进行了全面的研究,通过实验对现有的基准视频段落字幕数据集和建议EMVPC。从不同的视觉字幕任务的最先进的方案进行了比较,在15个流行的指标,并总结了详细的客观和主观的结果。最后,本文对视频段落字幕技术存在的问题和未来的发展方向进行了展望。这一独特的视角有望推动视频段落字幕研究的进一步发展。
Short-video Propagation Influence Rating: A New Real-world Dataset and A New Large Graph Model
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Qihoo 360 AI lab(奇虎360 AI实验室)
Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World
University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
University of Chinese Academy of Sciences(中国科学院大学);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心);Huazhong University of Science and Technology(华中科技大学)
MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);Shandong Normal University(山东师范大学);Renmin University of China(中国人民大学);Ant Group(蚂蚁集团);Nanyang Technological University(南洋理工大学);University of Oulu(奥卢大学)
FCMBench-Video: Benchmarking Document Video Intelligence
AI Lab, Qifu Technology(启赋科技AI实验室);College of Future Information Technology, Fudan University(复旦大学未来信息学院);School of Future Technology, South China University of Technology(华南理工大学未来技术学院);Pazhou Lab(琶洲实验室)
From Image to Video, what do we need in multimodal LLMs?
Intelligent Computing and Machine Learning Lab, School of ASEE, Beihang University(北京航空航天大学自动化学院智能计算与机器学习实验室);Xiaohongshu(小红书);School of Sino-French Engineer, Beihang University(北京航空航天大学中法工程师学院);College of Engineering and Computer Science, VinUniversity(Vin大学工程与计算机科学学院)
GoMatching: A Simple Baseline for Video Text Spotting via Long and Short Term Matching
Institute of Artificial Intelligence, Wuhan University(武汉大学人工智能研究院);College of Computing & Data Science at Nanyang Technological University(南洋理工大学计算与数据科学学院)
A Culturally-diverse Multilingual Multimodal Video Benchmark & Model
Mohamed bin Zayed University of Artificial Intelligence(莫德赫·本·扎耶德人工智能大学);University of Central Florida(中央佛罗里达大学);Islamic University of Technology(伊斯兰技术大学);Air University(空军大学);ETH Zurich(苏黎世联邦理工学院);Technische Universität München(慕尼黑技术大学);National Institute of Informatics(国家信息研究所);Australian National University(澳大利亚国立大学);Linköping University(利尔贝里大学)
ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
Peking University Shenzhen Graduate School(北京大学深圳研究生院);Pengcheng Laboratory(鹏城实验室);University of Oxford(牛津大学);Oxford-Suzhou Centre for Advanced Research(牛津-苏州高等研究院);Tencent AI Lab(腾讯AI实验室)